Special Review
% PRO^
,
# . -—¦ * office of Inspector General
EPA's Progress in Using the
Government Performance and Results Act
to Manage for Results
2001-B-000001
June 13, 2001
Accoumaniifty
Priorities
Strategies
People
Measurement

-------
Table of Contents
Executive Summary	 1
1.	Goals - Are EPA's goals consistent with its statutory
authority and key partner and stakeholder expectations? 	 3
2.	Priorities - Does EPA have a system for setting priorities
to best achieve its environmental mission?	 6
3.	Strategies - Has EPA developed results-oriented strategies
that clearly link annual plans and budgets to accomplishments?	 9
4.	Measurement - Does EPA have systems to measure,
evaluate, and report on environmental results?	14
5.	People - Does EPA have the human resource systems
needed to manage for results?	17
6.	Accountability - Has EPA established an accountability
system to ensure efficient achievement of its goals?	20
Exhibit 1: Scope and Methodology	25

-------
Executive Summary
The Government Performance and Results Act of 1993 (GPRA) seeks to shift the focus of
government performance and accountability away from simply the activities being performed to
the results and outcomes of those activities. The purpose of GPRA is to systematically hold
federal agencies accountable for achieving program results. To address GPRA requirements, the
U.S. Environmental Protection Agency (EPA) has taken a number of steps to implement a
practical GPRA framework to link EPA's activities to long-term results. EPA is considered a
leader in implementing GPRA because of the innovative approach it has taken to align its
planning, budgeting, analysis and accountability processes.
To determine what progress EPA has made in using GPRA to manage for environmental results,
we reviewed EPA's GPRA implementation activities, focusing on choices made, progress
achieved, challenges faced, and opportunities for improvement. We used the following six
questions to assess EPA's progress in implementing GPRA. We selected these six questions
because we believe they address areas that are critical in establishing an effective framework for
maximizing environmental results.
1.	GOALS - Are EPA's goals consistent with its statutory authority and key partner and
stakeholder expectations?
EPA's goals are consistent with traditional interpretations of its authorizing statutes and
GPRA, and accurately reflect the Agency's role in setting and implementing environmental
standards. However, EPA's goals have fallen short of key partner and stakeholder
expectations.
2.	PRIORITIES - Does EPA have a system for setting priorities to best achieve its
environmental mission?
While EPA sets multi-year, Agency level planning and budgeting priorities, there is no
readily visible system or process used to identify them. There is insufficient stakeholder
input and formal examination of options, costs, and benefits.
3.	STRATEGIES - Has EPA developed results-oriented strategies that clearly link
annual plans and budgets to accomplishments?
EPA has taken many actions to comply with GPRA's strategic planning requirements,
although it faces significant challenges in developing results-oriented strategies that clearly
link annual plans and budgets to accomplishment of long term outcomes. EPA needs to
better describe how, when, and at what cost agency goals are to be achieved; the
behavioral changes needed to accomplish these goals; and the links between current
activities and long term outcome goals.
4.	MEASUREMENT - Does EPA have systems to measure, evaluate, and report on
environmental results?

-------
EPA has taken steps to put systems in place to measure and report on progress under
GPRA, but continues to experience difficulty developing and obtaining the information
necessary for measuring, evaluating, and reporting on long-term environmental outcomes.
Most of EPA's environmental quality information is collected by states, and the
inconsistency and incompatibility of this data makes measurement difficult.
5.	PEOPLE - Does EPA have the human resource systems needed to manage for results?
EPA recognized the importance of human capital as a key Agency priority in its Strategic
Plan. However, EPA did not specifically address human capital in connection with any of
its goals, and needs to better integrate human capital into its overall strategic plan, develop
competencies in a wide range of skills, and better align individual accountability.
6.	ACCOUNTABILITY - Has EPA established an accountability system to ensure
efficient achievement of its goals?
EPA has established an accountability system with the intent of ensuring the efficient
achievement of its goals, and has been recognized as a leader in developing a goals-based
budget aligned with its programmatic and operational outputs and outcomes. However,
this framework tends to perpetuate EPA's "stove piped" organization, and does not
address overlapping environmental issues and the different needs and priorities of EPA's
various partners (states, local, tribal agencies).
In general, to improve GPRA implementation, EPA needs to strengthen its partnerships with
states and other federal agencies. Also, EPA needs to place greater focus on the ultimate results
and outcomes of its activities rather than actions performed. For example, EPA needs to more
carefully consider science and cost, in addition to laws and public perceptions, when setting
Agency priorities. Furthermore, EPA needs to invest in developing performance information that
is more outcome oriented.
2

-------
1. Goals
Are EPA's goals consistent with its statutory
authority and key partner and stakeholder
expectations?
EPA's 10 goals are consistent with traditional interpretations of its authorizing statutes and
GPRA, and accurately reflect the Agency's role in setting and implementing environmental
standards. However, EPA's goals have fallen short of key partner and stakeholder expectations in
several ways. Stakeholders have stated that EPA's media-
based goals restrict cross-media implementation, often the
preferred strategy for addressing local environmental
problems. In addition, stakeholders expressed concern that
EPA's non-environmental goals do not appear to be
consistent with GPRA's outcome orientation. Finally, some
of the original stakeholder expectations for EPA goals went
beyond EPA's authority under its environmental statutes and
GPRA.
Development of EPA Goals
EPA began developing national environmental goals in 1992,
a year before enactment of GPRA. The objective of this
effort, known as the National Environmental Goals Project,
was to develop national, cross-media goals that reflected the
priorities of the American public. This emphasis on public
priorities shaped the process for selecting goal topics. In
1994, EPA initiated a series of public meetings to obtain
public views. Public comment reflected various themes,
including the following:
Goals should cut across administrative boundaries of
EPA offices and other government departments.
Goals set at the national level need to be flexible enough to accommodate regional differences
and local priorities.
EPA used public input from the Goals Project to draft its goals, and initially developed
13 environmental goals. By condensing and combining some of the goals, EPA subsequently
reduced that number to seven environmental goals (Goals 1-7), and added three
non-environmental goals (Goals 8-10). These 10 goals formed the basis of EPA's 1997 Strategic
Plan.

EPA's 10 Goals
1.
Clean Air
2.
Clean and Safe Water
3.
Safe Food
4.
Pollution Prevention and Reducing

Risk in Communities, Homes,

Workplaces, and Ecosystems
5.
Better Waste Management,

Restoration of Contaminated

Waste Sites, and Emergency

Response
6.
Reduction of Global and Cross

Border Environmental Risks
7.
Expansion of Americans' Right to

Know About their Environment (a)
8.
Sound Science, Improved

Understanding of Environmental

Risk, and Greater Innovation to

Address Environmental Problems
9.
A Credible Deterrent to Pollution

and Greater Compliance Wth the

Law
10.
Effective Management
(a) in EPA's Revised Strategic Plan Goal 7 is titled
"Quality Environmental Information."
3

-------
EPA Emphasis on Traditional Role During Goal Development
EPA used three major criteria in 1992 to begin its goal development. First, the goals should be
environmental, so as to be meaningful and understandable by the general public. Second, the
goals should be attributable to the laws, programs, and resources that EPA administers, but broad
enough to encourage substantial roles by other parties. Third, the goals should be measurable,
using available data and systems, although data quality and availability should not entirely drive
the choice of goals. These three criteria led to goals that:
Paralleled EPA's organizational structure in many cases, reflecting a combination of
environmental media (air and water) and programs (waste management).
Emphasized progress toward compliance with health- and ecological-based environmental
standards established in EPA regulations (for example, attainment of air quality standards).
Focused mainly on EPA's role and statutory responsibilities rather than on a broader scope of
federal environmental responsibilities.
Mindful that GPRA directed agencies to develop strategic goals that reflect the agencies'
statutory authorities, EPA developed goals that accurately reflect its role in setting federal, health-
and ecological-based environmental standards and implementing and enforcing major federal
environmental laws. However, these goals have fallen short of key partner and stakeholder
expectations in several ways, as follows.
Cross-Media Implementation
In 1996, EPA chose a 10-goal structure composed of 7 goals derived from the National
Environmental Goals Project and 3 additional non-environmental goals. This goal structure
reflected existing Agency programs and structure and required less change to linkages among
existing planning, budget, and accountability systems. In doing so, the Agency chose not to adopt
a broader, more multi-media oriented goal structure proposed later in the process by EPA Region
5. According to EPA decision documents, this approach would have better supported place-
based and multi-media initiatives. EPA placed this more multi-media orientation in the Agency's
mission statement rather than in the specific goals and objectives used to track EPA's efforts.
EPA regions and the states have commented that EPA's 10-goal structure can restrict cross-
media implementation, often the preferred strategy for addressing local environmental problems.
Non-Environmental Goals
EPA expanded its goal structure by adding three non-environmental goals (Sound Science,
Enforcement, and Effective Management) to provide an explicit, comprehensive system of budget
accountability under GPRA for each of EPA's major program offices. EPA stakeholders,
however, have expressed concern that the non-environmental goals were process rather than
outcome oriented. In particular, states objected to EPA's enforcement goal as inconsistent with
4

-------
GPRA's outcome orientation. States have consistently commented that enforcement is a key tool
supporting federal and state environmental efforts in achieving environmental goals, but should
not in itself be considered an environmental goal.
Expectations Beyond EPA's Authority
To accomplish its goals and objectives, EPA works in partnership with many other federal
agencies. This includes agencies on whom EPA relies as co-implementors of federal
environmental programs (such as the Army Corp of Engineers) and as members of the regulated
community (such as the Departments of Energy and Defense). EPA also works with federal
agencies involved with public health and ecological protection (such as the Public Health Service
and Department of the Interior).
EPA's goals are generally consistent with GPRA's requirements limiting agency goal
development to areas of direct statutory responsibility. However, as stakeholders noted in the
National Environmental Goals Project, the ultimate aims of environmental laws — i.e., healthy
people and ecosystems — and the federal partnerships necessary to achieve those larger goals, are
beyond EPA's scope of authority or responsibility under GPRA. If EPA were to attempt to meet
these larger stakeholder expectations in the future, it would be necessary for EPA to seek the
support of the Administration and Congress for developing broader goals in partnership with
other federal agencies.
Recommendation
EPA has laid a solid foundation for future improvements with its goal development process.
There is a substantial record of stakeholder and partner opinion and preference. Stakeholders are
in substantial agreement over the importance of EPA goals for setting Agency direction and
strengthening federal and state partnerships. In order for the Agency to assume a greater
leadership role through the future development of its GPRA goals, we recommend that EPA:
1-1. Move expeditiously to amend EPA's goals and the processes for implementing them, as
necessary, to focus on environmental outcomes and encourage cross-media implementation.
EPA should also seek the support of the Office of Management and Budget and Congress
for a federal government process to develop environmental goals that promote the ultimate
human health and ecological aims of environmental laws, and the federal partnerships
necessary to achieve them.
5

-------
2. Priorities
Does EPA have a system for setting
priorities to best achieve its environmental
mission?
While EPA sets multi-year, Agency level planning and budgeting priorities, there is no readily
visible system or process used to identify them. Also, while EPA does consult informally with
stakeholders on planning and budgeting priorities, there is not a formal opportunity for comment
until submission of the budget to Congress.1 The priorities do not undergo any formal
examination of options, costs, and benefits. As a consequence, EPA's multi-year priorities have
been shaped more by law and public perception than by science and cost.
System for Setting Priorities
Because resources are limited, EPA must set priorities among competing environmental problems,
programs, and issues. EPA's media program offices set and implement priorities among the many
objectives in EPA's strategic plan by setting 5-year performance goals and allocating funds to
them annually. To set priorities, program offices consider information on legal authority, cost,
environmental risk, and public perceptions. However, we found that program offices do not
document or explain how these factors are used in decision making.
In discussions with Agency staff, we
found that the specific statutory
requirements of EPA's laws, as well as
public perceptions, have shaped EPA
priorities more than science and cost
information. EPA's senior management
has not established risk based decision-
making criteria to set priorities for the
Agency. Although environmental risk
assessment is used by EPA to set specific
health-based standards, it is not usually
used for setting broad Agency and
programmatic priorities.
Also, while EPA estimates the compliance costs of specific rules to assess cost-effectiveness,
there is no formal process for considering cost or cost effectiveness in Agency priority-setting
decisions. Thus, there is little assurance that EPA is allocating its limited resources to those
problems which pose the greatest environmental risks and opportunities for risk reduction.
1 OMB Circular A-ll Section 36, Communications with Congress and the Public and Clearance
Requirements, states, in part, that the nature and amounts of the President's decisions and the underlying materials
are confidential and are not to be released outside the Agency until the budget is transmitted to Congress.
Historically, EPA's priorities have been shaped by
a variety of factors:
Environmental laws
N,
Public perceptions of *
environmental problems^v
/
Science/risk information*


Internal EPA
Priority Setting
EPA Priorities
Economic costs*


6

-------
Consistency of Priorities
Each year, EPA identifies the Administrator's planning and budgeting priorities in budget
formulation guidance to EPA program offices. We identified 22 distinct Administrator priorities
in a review of Congressional budget summaries for fiscal years 1996 through 2000, covering
EPA's experience prior to and in the first year of GPRA implementation. Of these 22 priorities,
only 2 appeared in all 5 years, and 11 appeared only once or twice:
Administrator Priorities
Budget Years in Which Appear
Global Climate Change
Strengthen State and Tribal Partnerships
Improve Environmental Information
Brownfields
Achieve Progress on Clean Air Programs
Protect Children's Health
Reinvent Environmental Regulation
Safe Drinking Water
Toxic Waste Sites
Clean Water
Sound Science
Livable Communities
Toxic Pollutants
Community-Based Environmental Protection
Use Advanced Technologies in Environmental Management
Eliminating Barriers to Regulatory and Permitting Process
Implement Resource Conservation and Recovery Act Reinvention
Balanced Compliance and Enforcement
Implement Performance Partnership Agreements
Protect Human Health by Reducing Environmental Threats
Achieving Savings Through Streamlining efforts
Establish Working Capital Fund
Number of
Budgets
The multitude of national priorities is a source of concern and confusion to EPA's regions and the
states, as noted in a recent National Academy of Public Administration report:
EPA 's management system is stymied by a wide variety of priorities imposed by Congress and
the administration, and by itself. EPA staff are bewildered by the number of conflicting
priorities, statutory deadlines, court-imposed requirements and public participation needs.
States have challenged EPA to limit its priorities and devise a more flexible and open priority
setting approach that accounts for state priorities.
7

-------
Recommendation
By making EPA's priority setting clearer, the Agency can increase involvement of both internal
and external stakeholders. A more deliberate process for considering state priorities may begin to
address the criticism of EPA's national strategic planning process. In general, although risk and
cost information is imperfect, it is already used by some EPA program offices to set priorities.
Therefore, we recommend that EPA:
2-1. Define and document a national, multi-year priority setting process, and ensure that the
process fully and visibly uses available risk and cost information in setting priorities.
8

-------
o Q4ratpfiipc Has EPA developed results-oriented
o. oirdiegieb strategies that clearly link annual plans
and budgets to accomplishments?
EPA has taken many actions to comply with GPRA's strategic planning requirements, although it
faces significant challenges in developing results-oriented strategies that clearly link annual plans
and budgets to accomplishment of long term outcomes. EPA has provided a GPRA framework,
guidance, and training, among other actions. However, EPA needs to better describe how, when,
and at what cost agency goals are to be achieved; the behavioral changes needed to accomplish
these goals; and the links between current activities and long term outcome goals.
Agency Progress in Strategic Planning
Since GPRA's passage, EPA has undertaken many actions
to comply with the Act's strategic planning requirements
and enhance the Agency's ability to manage for results. A
critical early step was the establishment of a framework
within the Office of the Chief Financial Officer (OCFO)
designed to align planning, budgeting, analysis, and
accountability in an integrated system. This framework is
designed to link EPA's annual planning and budgeting
decisions to EPA goals.
In addition, OCFO has: (1) provided guidance on the
planning, budgeting, analysis, and accountability system;
(2) facilitated workgroups to help implement the new
system; (3) issued guidance on how to link annual
performance goals to long term outcomes; and (4) assisted
national program managers in developing and delivering
GPRA training to selected staff.
Another important step involved creating categories to aid
program planners in understanding where their environmental program goals and activities fall
within a results-oriented hierarchy. Known as the Hierarchy of Performance Indicators, program
managers are now required to use this environmentally-based categorization tool when developing
their annual performance goals and corresponding performance measures. According to OCFO
guidance, this tool should help the Agency move away from activity-based outputs to results-
based outcomes. The most recent categorizations from OCFO are shown in the following table.
Modified Hierarchy of Performance Indicators
Framework for GPRA
Implementation at EPA
(ioil Is
Objectives
Subob jcctivcs
Strategics
And IS u dieting
A n it u a I
Performance Goals
9

-------
Level
Measure of Progress
1A
Customer Satisfaction
1B
Service Delivery
1C
Any EPA, state/tribal, other governmental, multi-governmental, and/or multinational regulations,
activities, tools, standards
2
Actions/Responses by regulated community
3
Changes in, or prevention of, volume, mass, and/or toxicity of discharges, loadings, or emissions
4
Ambient conditions
5
Exposure or body burden/uptake
6
Risk or impacts to ecology, health, and/or welfare
R
Research and development
Agency Challenges in Strategic Planning
Although the actions taken by EPA to comply with GPRA's strategic planning requirements have
helped managers improve their focus on results, the Agency still faces a number of challenges.
These include: (1) describing in sufficient detail for external stakeholders how, when, and at what
cost goals are to be achieved; (2) ensuring that strategies clearly identify the behavioral changes
required to achieve the results-oriented outcomes desired; and (3) clearly linking current activities
to long term outcome goals.
Describing Plans in Sufficient Detail
Our analysis of the key components of five key GPRA strategic planning documents2 for three
EPA goals (air, water, and waste management) indicated that these strategies do not provide
sufficient detail to allow meaningful evaluation and input by interested stakeholders for the out
years. In each instance, the strategies were vague in their explanation of the actions that state,
local, and tribal agencies needed or would be expected to take to achieve the program's outcome
goals. Congress anticipated that it may take several planning cycles to refine the strategic
planning process and that strategic plans would be continually improved as various planning
cycles occurred. According to the General Accounting Office's (GAO's) 1997 GPRA Guide,3
external reviewers of agency strategies should be able to understand and evaluate the linkages
between inputs, outputs, and outcomes and reasonably see which activities lead to what
intermediate and, eventually, long term outcomes. Vagueness in this critical aspect of the
2
The documents are (1) FY 1999 Annual Performance Report - EPA's March 2000 GPRA submission to Congress (Version ID 3830
enacted in President's Budget, Version 1); (2) EPA's September 2000 Strategic Plan (EPA/190-R-00-002); (3) EPA's September 1997 Strategic Plan
(EPA/190-R-97-002); (4) EPA's FY 2001 Annual Plan (EPA/205-R-00-002); and (5) EPA's FY 2001 Congressional Justification (CJ), also known
as the Budget Justification.
3
Agencies' Strategic Plans Under GPRA: Key Questions to Facilitate Congressional Review, GAO/GGD-IO.1.16, May 1997.
10

-------
agency's GPRA planning was previously noted by the House Science Committee in 1997 when it
examined EPA's first plan. The committee noted that
The EPA draft plan acknowledges the States as a significant partner in achieving
environmental goals. Yet beyond this acknowledgment, the State role (as well as
that of the Indian tribes) is unaddressed in the draft strategic plan.
Desire for greater understanding of the details about how EPA plans to achieve its GPRA goals
continues to be expressed by the public, members of Congress, GAO, and others. EPA officials
for air, water, and waste management recognize that their respective programs need to improve
their descriptions of how goals are to be achieved, when, and at what cost, particularly for
Years 2 through 5 of their strategies. They also pointed out that, in some instances, they have
developed more detailed strategies which could be cost effectively linked using the World Wide
Web. For example, air program officials noted that they developed a fairly extensive urban air
toxics strategy in 1999, setting forth the agency's strategic thinking 10 years into the future.
Similarly, water program officials noted that in 1998 they developed a clean water action plan
describing over 100 actions that EPA, Agriculture, and other federal agencies planned to take to
improve the Nation's surface water over a 7-year period. Lastly, EPA officials suggested that the
existing Hierarchy of Performance Indicators could be improved by establishing a category for
state, local, and tribal partners separate from EPA programs.
Ensuring Clear Identification of Behavioral Changes
A second challenge involves ensuring that strategies clearly identify the actions and behavioral
changes that the regulated community needs or would be expected to make in order for the
Agency to achieve its outcome goals. Our analysis of the GPRA strategic planning documents
indicates that the air and water program strategies do not provide sufficient details about the
expectations for the regulated community. The strategies were vague in their explanation of the
actions that the regulated community would need to take to achieve the program's outcome
goals. The following table shows how these behavioral changes are a critical step in the logic
matrix for accomplishing EPA's outcome goals.
11

-------
Logic Matrix for Accomplishing Goals
Note a
Note b
Note c
INPUTS
OUTPUTS
OUTCOMES
~LT
Level 1
Level 2
Level 3
Level 4 | Level 5 | Level 6
Internal
EPA
Actions arid
Activities
Actions by
Actions and
Behavioral
Changes
by the
Regulated
Community
Reduced
Amounts
and/or
Toxicity of
emissions
Improved
ambient
environ -
mental
conditions
Changes in
ambient
conditions
relative to
health-based
standards
Reduced
exposure
or body
burden
Human and
ecological
health
Loca
Triba and
other
Regulatory
Agencies
Objectives
Subobjectives
Strategies
Annual Perform
ance Goals
Progress toward
long-term outcomes
Annual Planning
and Budgeting
Note a: Simple logic model categorizations.
Note b: OCFO Modified Hierarchy of Indicators
Note c: EPA/OIG expanded hierarchy of indicators
According to representatives of the national programs for air and water, using logic model
concepts to perform a matrix analysis similar to that above would be helpful in understanding the
critical linkages in managing for results. They noted that it is difficult to link behavioral
expectations in their media programs when the majority of activities and funding for the needed
actions are contained in a separate Agency goal. Although a critical link in the GPRA process,
they explained that the existing Agency goal structure, where compliance and enforcement is a
separate goal (Goal 9) generally funded and carried out separately from their media programs,
sometimes hinders their efforts to clearly identify the actions and behavioral changes that the
regulated community needs or would be expected to make in order for the Agency to achieve its
outcome goals. Representatives of both programs noted that EPA's decision to separate
compliance and enforcement goals from media program goals reflected extensive deliberations
and recognized that there would be tradeoffs; however, they also said it may be appropriate to
evaluate these tradeoffs again.
Linking Current Activities to Long Term Outcome Goals
A third challenge the Agency faces involves clearly linking current activities to long term outcome
goals. A vague statement of actions beyond the near term makes it difficult for external groups to
see clear linkages, although the agency staff may be well aware of these linkages. Additionally,
the Agency's overall long term goal - - true human and ecological health - - is shared with many
12

-------
other federal and non-federal entities, difficult and costly to measure directly, and hard to
explicitly link to some near term agency activities. As such, strategies often rely heavily on the
fulfillment of largely EPA-controlled surrogate measures of human and ecological health, such as
a reduction in the emissions of regulated pollutants or a reduction in the number of violations
reported annually. The implicit assumption is that these reductions or changes will eventually lead
to true human and ecological health.
EPA's strategies recognize the need to rely on others for long term goal achievement. However,
EPA needs to provide more explicit descriptions of the partnering activities, roles, and
responsibilities for outcomes by these non-EPA groups. This would better enable external
reviewers to both understand and evaluate the linkages between inputs, outputs, and outcomes,
and accomplish GPRA's purpose of improving the public's confidence in the ability of federal
agencies to manage for results.
Recommendations
We believe EPA has an opportunity to improve its GPRA strategic plan and simultaneously
improve its relationship with state, local, and tribal agencies, as well as improve the public's
confidence in the Agency. As such, we recommend that EPA:
3-1. Link other, more detailed, planning documents to its GPRA strategic plan by providing
World Wide Web citations (links), thus providing external stakeholders with easy access
to more in-depth information on how EPA plans on achieving long term goals.
3-2. Revise the existing Hierarchy of Performance Indicators to clearly establish a place for
state, local, and tribal agencies' expected roles in achieving environmental goals and
proactively involve these partners in GPRA strategic planning for the next round of
strategic plans, due in 2003.
3-3. Work across agency goals to develop clear expectations for the behavioral changes EPA
expects industry and the regulated public to make if envisioned goals are to be achieved.
3-4. Work with other federal agencies that have human health and ecology goals to establish
long-term outcome-oriented goals that reflect the coordinated efforts of multiple federal
agencies.
13

-------
4. Measurement
Does EPA have systems to
measure, evaluate, and report on
environmental results?
EPA has taken steps to put systems in place to measure and report on progress under GPRA, but
continues to have difficulty developing and obtaining the information necessary for measuring,
evaluating, and reporting on long-term environmental outcomes. Since GPRA's enactment, EPA
has aligned its management systems to help support implementation of GPRA, by establishing:
An Office of the Chief Financial Officer, to work with program offices to develop and track
Agency-wide environmental measures.
• An Office of Environmental Information, to centralize responsibility for improving interagency
data sharing, as well as improve the accuracy, reliability, and scientific basis of environmental
information.
An Evaluation Network, to stimulate and promote evaluation as a valuable tool for EPA
program managers.
EPA has been recognized for its effort to align its budgeting, planning, and accounting systems in
an attempt to track and report on the resources used in achieving its GPRA objectives.
Nonetheless, EPA continues to face challenges in implementing GPRA. EPA's performance
information tends to focus on program activities (outputs) rather than results (outcomes), due to
the difficulty in obtaining and measuring environmental results. Further, the majority of EPA's
information is collected by states, and the inconsistencies in that information makes its use for
measuring, evaluating, and reporting difficult.
Measuring and Reporting on Results
EPA faces a number of challenges in attempting to
measure and report on environmental results. A
majority of EPA's annual performance goals and
measures focus on program activities or outputs
(number of enforcement actions, number of permits
issued, etc.). Despite the vast array of data in EPA's
information systems, GAO, the states, regulated
entities, and EPA itself have pointed out that the
Agency does not have much of the information it
needs pertaining to environmental conditions and
trends and the potential human health risks of various
pollutants. This makes it difficult to report on the
environmental results of EPA's activities. Further,
environmental programs may take several years to
Availability of Environmental
Results Data
Activity/Output
Measures
Human and
Ecological Health
Outcomes	
Data Unavailable at
present Time
14

-------
show results. Thus, measuring and reporting on end results may also take years. In addition,
even with better measures and data, linking environmental programs and activities to outcomes is
complicated by a myriad of external factors, including weather, international environmental issues,
and economic activity, many of which are outside of EPA's control.
Further, while EPA does not currently receive all state information, a majority of the information
that EPA uses for measuring and reporting is collected by states. According to the Environmental
Council of States, some 94 percent of all environmental quality data in national databases
originates with the states. In previous reports, the OIG, as well as others, have found data for
individual states to be inconsistent, and data among various states to be incompatible. States are
encouraging EPA to allow more flexibility in what data is collected and how it is collected.
According to the Environmental Council of States, the states would rather concentrate on setting
broad indicators of environmental improvement and state capacity rather than on traditional
statistical and financial measures. This concern is echoed by the regulated community, which is
urging EPA to focus on defining, measuring, and rewarding environmental results and reorienting
core regulatory functions so they are driven primarily by performance goals and not activity
measures. While the EPA's Office of Chief Financial Officer continues to work with program
offices to develop measures that demonstrate environmental outcomes, it is proving to be a
difficult and complex task.
Evaluating and Reporting on Results
There is an important distinction between performance measurement and program evaluation.
Measurement is the ongoing monitoring and reporting of program accomplishments and seeks to
answer the question, "what was achieved?" Program evaluation attempts to determine the
manner and extent to which programs achieve intended objectives, usually through objective
measurement and systematic analysis. The intent is to analyze why results were achieved, so that
further action can be taken based on lessons learned. Thus, evaluation is aimed at providing the
link between actions taken and results achieved.
According to GAO, in some programs, outcomes are not quickly achieved or readily observable,
or their relationship to the program is uncertain. In such cases, program evaluation is needed,
along with performance measurement, to examine the extent to which a program is achieving its
objectives.4 EPA has been without a centralized evaluation function since 1995, when the
Program Evaluation component of the Office of Policy, Planning and Evaluation was disbanded.
More recently, EPA established a Program Evaluation Network to help stimulate interest in
evaluation and increase the Agency's capacity to conduct evaluations. The Office of Policy,
Economics and Innovation created a division to evaluate EPA's many reinvention initiatives and
help support and promote evaluation activities throughout the Agency. OIG has also established
an Office of Program Evaluation to evaluate EPA programs.
4 Program Evaluation: Studies Helped Agencies Measure or Explain Program Performance
(GAO/GGD-00-204)
15

-------
While the Office of Chief Financial Officer's Office of Planning, Analysis and Accountability has
an analysis component, the staff in this office do not see their role as conducting evaluations for
EPA program offices. Rather, they view their role in evaluation as facilitating and helping EPA
programs develop capacity through mechanisms such as the evaluation network. EPA program
offices are beginning to focus some attention on conducting evaluations, but the resources they
are devoting are limited and the distribution of those resources uneven from program to program
Additionally, EPA has not established policy or guidance on how program evaluation results are
to be used and tracked by management. Without such controls, there is little assurance that
evaluative results will be used.
Recommendations
For EPA to demonstrate and report on the outcomes and impact of environmental programs, a
concerted effort will be needed between EPA, states, and the regulated community to develop
outcome-oriented performance information and environmental indicators. Where such
information does not exist or will take time to develop, program evaluation can help bridge the
gap and examine the extent to which programs are achieving their objectives. Consequently, we
recommend that EPA:
4-1. Work with states and the regulated community to develop an information strategy that
includes:
Clearly defined human health and ecological outcomes, as well as output measures.
Expanded use of environmental indicators and performance-based management.
Clear and consistent data definitions and standards for improving data quality.
Identification of the resources necessary to obtain, compile, store, and share data.
4-2. Encourage the use of evaluation staff as a resource for program managers and develop
guidance for how evaluative results should be used and tracked.
16

-------
c Ppnnlp Does EPA have the human resource systems
¦	^	needed to manage for results?
EPA recognized the importance of human capital as a key Agency priority in its Strategic Plan,
and has recently developed and begun implementing a separate Strategy for Human Capital.
However, in its Strategic Plan, EPA did not specifically address the human capital needs of any of
its strategic goals. A prerequisite for EPA's successful implementation of GPRA is the
application of the right skills and competencies within a results-oriented culture. EPA needs to
better integrate human capital into strategic plans; more effectively develop competencies in
management, science, and technical skills; and better align individual accountability with Agency
goals and results.
Human Capital Emphasized
GPRA expresses the importance of planning human capital needs to attain Agency goals. Several
independent evaluations by GAO, National Academy of Public Administration, Environmental
Council of States, National Research Council, and EPA's OIG have focused on aspects of EPA's
human capital management. These evaluations addressed the need for developing and aligning
scientific and management competencies, as well as individual performance expectations and
accountability, with strategic goals. For example:
GAO stated that, "successful human capital management strategies are the key to maintaining a highly
skilled, energized, and empowered workforce that is focused on results."
GAO also stated, "EPA is not able to identify the size of its workforce and the competencies that need
to be deployed among its organizational components to effectively carry out its strategic goals and
objectives."
The National Research Council stated that, "to achieve scientific and technical excellence, EPA must
attract, retain, and properly support first rate dedicated professional staff."
The OIG recently reported in Superfund that the Headquarters program office and several regions
did not identify the quality assurance training needs, and in some regions training was not always
provided. Also, audits have repeatedly noted a need to better train managers in their
administration and oversight of EPA's assistance agreements. Additionally, we found EPA
employees in the hazardous waste program needed more rigorous training to calculate proposed
penalties against violating facilities, and that the National Environmental Performance Partnership
System (NEPPS) has not been well-integrated into EPA due to lack of, among other things,
training and guidance. Human capital concerns will intensify in the coming years as 52% of
Senior Executive Service, 47% of scientific staff, and 25% of EPA's total workforce will be
eligible for retirement within 5 years, while competition for scientific, technical, and managerial
talent will likely increase.
17

-------
Human Resource Elements Essential
Effective implementation of performance-based management as envisioned by GPRA is dependent
on senior management's willingness and ability to strategically manage all of the Agency's
resources, including human capital. Specifically, as noted by GAO, this requires aligning and
developing day-to-day workforce competencies and expectations with strategic and program
plans and results at all levels for efficient and effective application of limited human resources, and
to promote a results-oriented culture. Human resource elements essential for successfully
implementing GPRA and attaining EPA's goals are:
Integrating Human Capital into Strategic Plans
Developing Competencies in Management, Science, and Technical Skills
Aligning Individual Accountability with Agency Goals and Results
Our review of these elements noted the following future challenges EPA faces in attaining its goal
of having the human resource systems needed to manage for results.
Integrating Human Capital into Strategic Plans
The Agency recognizes that one of its biggest challenges over the next several years is the
development and implementation of a workforce planning system that focuses attention on the
identification of needed skills and competencies, and the means of addressing them. EPA
developed a Workforce Development Study to discuss competencies needed to: meet the current
Agency mission; identify possible future missions of the Agency; and suggest new competencies
for future scenarios. EPA has broadly recognized the importance of human capital as a key
Agency priority in its September 2000 Strategic Plan, but has not specifically addressed human
capital needs in its goals. Although neither the Strategic Plan nor the Workforce Study address
specific human capital issues or make commitments to specific human capital actions, EPA is
developing a more specific workforce development plan.
EPA needs to follow through on its recently developed Strategy for Human Capital by making a
consistent commitment to evaluate the number of staff and competencies needed, and how those
employees should be deployed among its program areas and in various locations to meet its
strategic goals. GAO recently reported that while EPA has implemented several initiatives during
the past decade to better understand the demands facing its workforce, these initiatives have not
received the resources and senior management commitment to bring them to fruition. Therefore,
EPA needs to implement a strong management approach to training and development, and
organize its efforts around principles agreed upon by senior Agency management. The need for
developing, integrating, and aligning human capital planning with the strategic and annual plans
will be magnified as: EPA's role changes; the need for specific sophisticated scientific,
management, social, and technical skills increases; the current workforce ages; and competition to
recruit and retain highly qualified staff intensifies.
18

-------
Developing Competencies in Management, Science, and Technical Skills
One of EPA's long-standing human capital challenges is attaining and developing the needed staff
competencies in management, scientific, and technical positions. EPA has not assessed the gap
between (1) critical skills and competencies needed to achieve its goals and objectives and (2)
existing staff skills and competencies. Furthermore, appropriate training is essential for
management to lead and promote a results-oriented culture in EPA and for scientific and technical
staff to achieve and maintain excellence and credibility among their peers. However, National
Research Council, GAO and OIG reviews found that EPA staff often lacked specific training for
technical, scientific, and financial program responsibilities.
Aligning Individual Accountability with Agency Goals and Results
GPRA requires EPA to clearly establish results-oriented performance goals for which it will be
held accountable. Since Agency accountability is dependent upon the collective accountability of
its staff, EPA must align individual employee performance expectations with Agency goals. This
is necessary so that managers and staff understand the connection between their daily activities
and the Agency's progress in accomplishment of its goals. As described by EPA staff at different
levels and components, EPA has not consistently shifted the orientation of individual performance
expectations and accountability from process or activity completion to achieving or contributing
toward attainment of Agency goals. EPA also has not integrated a GPRA results-based
accountability framework and culture into staff day-to-day activities by regularly incorporating
results-based expectations into its staffing, performance management, and development
documents, such as vacancy announcements, position descriptions, performance agreements, and
individual development plans.
Recommendations
For EPA to achieve its strategic goals, it must focus on how to best invest in its people and link
their responsibilities and performance to Agency results. EPA is taking steps in the right direction
with its workforce development plans and strategies, but now must make decisive commitments
and investments to integrate human capital requirements into its Strategic Goals and Annual
Plans. In order for EPA to effectively implement GPRA, we recommend that EPA:
5-1. Identify needed scientific, technical, and management competencies to achieve specific
mission goals, and measure the extent to which employees possess these competencies.
5-2. Fund and implement training and development to address skill gaps.
5-3. Apply management, technical, and scientific competency criteria to employee recruitment,
selection, development, and performance accountability processes.
5-4. Align staff performance expectations and contributions with Agency goals.
19

-------
6. Accountability
Has EPA established an
accountability system to ensure
efficient achievement of its goals?
EPA has established an accountability system with the intent of managing resources efficiently,
with information to ensure the effective achievement of its goals, and has been recognized as a
leader in developing a goals-based budget aligned with its programmatic and operational outputs
and outcomes. However, this framework, while well intentioned, tends to perpetuate EPA's
"stove-piped" organization. Consequently, it does not address overlapping environmental issues,
as well as the different needs and priorities of EPA's partners - the states - who implement the
majority of EPA's programs. EPA's system of accountability for attainment of its goals do not
sufficiently link or align authority, responsibility, resources, measures, and outcomes.
Statutes Require Accountability
Congress' desire to hold agencies accountable for their performance and results was the motivating
force behind the Chief Financial Officers Act of 1990 (CFO) and GPRA. While the CFO Act
established the foundation for improving management and financial accountability, GPRA created
requirements for agencies to generate additional information Congressional and executive branch
decision makers need in considering measures to improve government and reduce costs. The
purpose of GPRA, as stated in the law, is to systematically hold federal agencies "accountable for
achieving program results." Additionally, the Reports Consolidation Act of 2000 authorizes
agencies to combine performance information with cost information, to impose a new, more
business-like framework for management and accountability for federal agencies. Appropriate
information on results and costs is necessary for accountability.
EPA Has Established an Accountability System
EPA has established an accountability
system tied to its goals, as shown in
the accompanying chart. The Agency
has taken a goals-based budget that is
aligned with its programmatic and
operational outputs and outcomes as
a means of accountability and to
accomplish its goals. EPA has been
recognized as a leader in developing
this innovative approach.
EPA's Accountability System
Annual Performance
Plan and Budget
I
Memoranda of Agreement

Grant
Workplans/Performance
Partnership Agreements
Administrator's
Priorities
National Program
Goals and Priorities
States and Tribes
Annual Performance
Report/ Mid-Year Deputy
Administrator Meetings
Regional Mid-Year and
Annual Reporting
Accomplishment Reports
20

-------
Essential elements for an accountability system that will ensure efficient achievement of EPA
goals are:
• Alignment of EPA Goals with Stakeholder Multimedia Priorities
Agreement on Roles, Responsibilities, and Expectations
Comparability and Standardization of Results-Oriented Measures and Data
Linkage of Cost Accounting Information with Activities used for Decision Making, and
Results
Our review of these elements noted the following challenges EPA faces in attaining its goal of
having an effective accountability system.
Alignment of EPA Goals with Stakeholder Multimedia Priorities
EPA establishes its long-term and annual goals and develops its budget by goal. Accountability
for progress and the use of resources by EPA goal is not consistent with the flexibility necessary
for multi-media activity, investment, and priorities that EPA regional offices need in order to
address the interests of state partners. For example, EPA regional officials told us that EPA's
goal structure tends to promote a stove-piped organization and that it is not conducive to cross-
media work, such as Environmental Justice, Brownfields, and Children's Health. EPA needs to
develop a process to accurately capture state priorities, and where agreed upon, adjust budgets
and accountability to reflect those priorities.
Agreement on Roles, Responsibilities, and Expectations
EPA depends heavily on states to fund and implement national programs as well as provide most
of the environmental data. The charts below demonstrate the trend in environmental work done
by the states in terms of percentage of the programs that have been delegated to the states, and
the relative percentage of EPA's investment in environmental protection compared with that of
the states. Also, according to the Environmental Council of States, at least 80 percent of all
environmental enforcement actions are taken by the states.
FY 1996 Environmental Investment
By States & U.S. EPA
$ in Billions
EPA
States
65.8%
Total: $19.0
Percent of Environmental Programs
Delegated to the States

71%

1998

41%



1993


Source: Environmental Council of States (ECOS)
21

-------
EPA and states have not yet agreed on how states should be provided with flexibility, along with
accountability, for environmental results. Relations between EPA and states have been strained
due to disagreements over:
Respective roles and the extent of federal oversight
Priorities and budgets
Results-oriented performance measures, milestones, and data
Comparability and Standardization of Results-Oriented Measures and Data
There is limited standardization and comparability of data between states. As already noted, the
Environmental Council of States estimates that about 94 percent of the national data on
monitoring of environmental quality is collected by the states, and lack of compatibility of data
makes it difficult to aggregate data for trending, comparing, and evaluating performance and
progress.
Linkage of Cost Accounting Information with Activities and Results
Since Fiscal Year 1999, EPA has been able to link its budget and costs to goals, objectives, and
sub-objectives, but not often to activities and outcomes. For example, EPA's FY 2000 Annual
Report does not provide cost information associated with its annual performance goals and
measures. According to the Office of the Chief Financial Officer, at least one EPA region is
piloting a cost accounting and performance reporting tool, and EPA has also adopted cost
accounting practices in the Superfund program, Oil Spill programs, and in the Working Capital
Fund. EPA has not consistently decided what level of cost accounting is needed for decision
making in all programs and has not defined an approach for using cost information. EPA's cost
accounting system generally does not track costs below the Sub-Objective (i.e., 1-5 year) level, nor
does it track across goals as demonstrated in the diagram below. The Agency recognizes the need
for cost accounting for key activities and is currently developing cost accounting tools to better
respond to Agency needs. To obtain this information, EPA should identify the key activities for
which it needs to accumulate costs and develop a means for collecting and using this information
for results based management, program execution, and evaluation.
(¦Hill

Sul>-()l>jeili\e
Strategy/Activity
Annual Performance
Goal
Hierarchy of EPA Resource Allocation
A cost accounting system linking activity costs with performance,
would enable EPA to:
Identify Return of Investment
Compare investment options and trade-offs
Account for many regional and state multi-media activities
linked to multiple sub-objectives
Promote and recognize greater operational efficiencies
Make optimal budgeting and planning decisions
Annual Planning &
Budgeting
22

-------
Recommendations
EPA has many federal and state partners and stakeholders that collectively have a variety of needs,
levels of involvement, and interdependence in contributing to EPA's national goals. EPA can
improve its accountability by seriously considering state and stakeholder priorities and
responsibilities in its planning, budgeting, and reporting systems, and aligning its goals, activities
and resources with performance measures and results. The Agency has the capacity to link its
budgeting, planning, and accountability information at a high level and must follow through on the
actions it has started to improve performance and cost data. To strengthen accountability
processes, we recommend that EPA:
6-1. In setting expectations:
Define roles and responsibilities for EPA and its partners;
Agree on priorities, strategies and results; and
Develop and use better environmental performance measures and data.
6-2. In the short-term, integrate existing cost and performance data, through Agency-wide cost
accounting, to:
Identify return on investment, both at the activity and outcome level;
Compare different investment options for future investment choices; and
Promote and recognize greater efficiencies and opportunities for savings.
6-3. In the long-term, identify the key activities for which it needs to accumulate costs and
develop a means for collecting and using this information for budget formulation and
planning, program execution, and evaluation.
23

-------
Exhibit 1
Scope and Methodology
Purpose of Review
To determine what progress EPA has made in using GPRA to manage for environmental results,
we conducted a review of GPRA's implementation activities. The intent was to inform EPA's
leadership and interested stakeholders about EPA's decisions that set the stage for GPRA
implementation; progress in using GPRA; challenges that EPA faces in implementing GPRA; and
opportunities for near and long-term improvements.
Questions Posed
To conduct our review, we posed questions related to six different critical aspects of GPRA:
1.	Goals - Are EPA's goals consistent with its statutory authority and key partner and
stakeholder expectations?
2.	Priorities - Does EPA have a system for setting priorities to best achieve its environmental
mission?
3.	Strategies - Has EPA developed results-oriented strategies that clearly link annual plans
and budgets to accomplishments?
4.	Measurement - Does EPA have systems to measure, evaluate, and report on
environmental results?
5.	People - Does EPA have the human resource systems needed to manage for results?
6.	Accountability - Has EPA established an accountability system to ensure efficient
achievement of its goals?
Review Steps Taken
To answer these questions, we interviewed EPA Headquarters and regional officials, and also
discussed GPRA implementation with environmental officials in five states. In addition, we
reviewed previous reports, publications, and correspondence related to GPRA implementation
written by the:
General Accounting Office
Environmental Council of States
Senate Governmental Affairs Committee
House Oversight and Reform Committee
24

-------
National Academy of Public Administration
National Research Council
George Mason University's Mercatus Center
We also took into account information in other reports published by the EPA OIG.
We reviewed Agency-wide projects on core competencies and suggestions by states and the
regulated community for improving EPA's management of environmental programs. We reviewed
the GPRA framework for EPA's Offices of Water, Air and Radiation, and Solid Waste and
Emergency Response (Goals 1, 2 and 5). We reviewed publicly available documentation, such as
EPA's Strategic Plan, Congressional Budget Justification, Annual Performance Plan, and Annual
Performance Report; as well as comments from the public, states, and the regulated community
regarding these documents. This review was considered a special project and, as such, was not
conducted in accordance with Generally Accepted Government Auditing Standards.
25

-------