f
of Inspector General
Special Review
EPA's Progress in Using the
Government Performance and Results Act
to Manage for Results
2001-B-000001
June 13, 2001
Goals
AccGEintatfifrty
Managing
For;
Resiriis/
People
Prlorftfes
Strategies
[EPA
350/
2001.5
U.S. EPA Headquarters Library
Mail code 3201
120° Pennsylvania Avenue NW
Washington DC 20460
-------
-------
Table of Contents
Executive Summary 1
1. Goals - Are EPA's goals consistent with its statutory
authority and key partner and stakeholder expectations?
2. Priorities - Does EPA have a system for setting priorities
to best achieve its environmental mission? .
3. Strategies - Has EPA developed results-oriented strategies
that clearly link annual plans and budgets to accomplishments? 9
4. Measurement - Does EPA have systems to measure,
evaluate, and report on environmental results? 14
6. People - Does EPA have the human resource systems
needed to manage for results? 17
6. Accountability - Has EPA established an accountability
system to ensure efficient achievement of its goals? 20
Exhibit 1: Scope and Methodology . . . 25
-------
-------
Executive Summary
The Government Performance and Results Act of 1993 (GPRA) seeks to shift the focus of
government performance and accountability away from simply the activities being performed to
the results and outcomes of those activities. The purpose of GPRA is to systematically hold
federal agencies accountable for achieving program results. To address GPRA requirements, the
U.S. Environmental Protection Agency (EPA) has taken a number of steps to implement a
practical GPRA framework to link EPA's activities to long-term results. EPA is considered a
leader in implementing GPRA because of the innovative approach it has taken to align its
planning, budgeting, analysis and accountability processes.
To determine what progress EPA has made in using GPRA to manage for environmental results,
we reviewed EPA's GPRA implementation activities, focusing on choices made, progress
achieved, challenges faced, and opportunities for improvement. We used the following six
questions to assess EPA's progress in implementing GPRA. We selected these six questions
because we believe they address areas that are critical in establishing an effective framework for
maximizing environmental results.
1. GOALS - Are EPA's goats consistent with its statutory authority and key partner and
stakeholder expectations?
EPA's goals are consistent with traditional interpretations of its authorizing statutes and
GPRA, and accurately reflect the Agency's role in setting and implementing
environmental standards. However, EPA's goals have fallen short of key partner and
stakeholder expectations.
2. PRIORITIES - Does EPA have a system for setting priorities to best achieve its
environmental mission?
While EPA sets multi-year, Agency level planning and budgeting priorities, there is no
readily visible system or process used to identify them. There is insufficient stakeholder
input and formal examination of options, costs, and benefits.
3. STRATEGIES - Has EPA developed results-oriented strategies that clearly link
annual plans and budgets to accomplishments?
EPA has taken many actions to comply with GPRA's strategic planning requirements,
although it faces significant challenges in developing results-oriented strategies that
clearly link annual plans and budgets to accomplishment of long term outcomes. EPA
needs to better describe how, when, and at what cost agency goals are to be achieved; the
behavioral changes needed to accomplish these goals; and the links between current
activities and long term outcome goals.
-------
4. MEASUREMENT - Does EPA have systems to measure, evaluate, and report on
environmental results?
EPA has taken steps to put systems in place to measure and report on progress under
GPRA, but continues to experience difficulty developing and obtaining the information
necessary for measuring, evaluating, and reporting on long-term environmental outcomes.
Most of EPA's environmental quality information is collected by states, and the
inconsistency and incompatibility of this data makes measurement difficult.
5. PEOPLE - Does EPA have the human resource systems needed to manage for results?
EPA recognized the importance of human capital as a key Agency priority in its Strategic
Plan. However, EPA did not specifically address human capital in connection with any
of its goals, and needs to better integrate human capital into its overall strategic plan,
develop competencies in a wide range of skills, and better align individual accountability.
6. ACCOUNTABILITY - Has EPA established an accountability system to ensure
efficient achievement of its goals?
EPA has established an accountability system with the intent of ensuring the efficient
achievement of its goals, and has been recognized as a leader in developing a goals-based
budget aligned with its programmatic and operational outputs and outcomes. However,
this framework tends to perpetuate EPA's "stove piped" organization, and does not
address overlapping environmental issues and the different needs and priorities of EPA's
various partners (states, local, tribal agencies).
In general, to improve GPRA implementation, EPA needs to strengthen its partnerships with
states and other federal agencies. Also, EPA needs to place greater focus on the ultimate results
and outcomes of its activities rather than actions performed. For example, EPA needs to more
carefully consider science and cost, in addition to laws and public perceptions, when setting
Agency priorities. Furthermore, EPA needs to invest in developing performance information that
is more outcome oriented.
-------
1. Goals
Are EPA's goals consistent with its statutory
authority and key partner and stakeholder
expectations?
EPA's 10 goals are consistent with traditional interpretations of its authorizing statutes and
GPRA, and accurately reflect the Agency's role in setting and implementing environmental
standards. However, EPA's goals have fallen short of key partner and stakeholder expectations
in several ways. Stakeholders have stated that EPA's media-
based goals restrict cross-media implementation, often the
preferred strategy for addressing local environmental
problems. In addition, stakeholders expressed concern that
EPA's non-environmental goals do not appear to be
consistent with GPRA's outcome orientation. Finally, some
of the original stakeholder expectations for EPA goals went
beyond EPA's authority under its environmental statutes and
GPRA.
'8.
Development of EPA Goals
EPA began developing national environmental goals in
1992, a year before enactment of GPRA. The objective of
this effort, known as the National Environmental Goals
Project, was to develop national, cross-media goals that
reflected the priorities of the American public. This
emphasis on public priorities shaped the process for
selecting goal topics. In 1994, EPA initiated a series of
public meetings to obtain public views. Public comment
reflected various themes, including the following:
Goals should cut across administrative boundaries of
EPA offices and other government departments.
Goals set at the national level need to be flexible enough to accommodate'regional
differences and local priorities.
EPA used public input from the Goals Project to draft its goals., and initially developed
13 environmental goals. By condensing and combining some of the goals, EPA subsequently
reduced that number to seven environmental goals (Goals 1-7), and added three
non-environmental goals (Goals 8-10). These 10 goals formed the basis of EPA's 1997 Strategic
Plan.
iEPA's 4<) Goals
: 1.- ;,Clean Air , ... -, ....
2. -' Ctean and £afe Water .
" SafeWood - * r -:<-,-.-. ;;v \
Pollution Prevention;and "
Reducing Risk in Communities,
'Homes; Workplaces, ah
-------
EPA Emphasis on Traditional Role During Goal Development
EPA used three major criteria in 1992 to begin its goal development. First, the goals should be
environmental, so as to be meaningful and understandable by the general public. Second, the
goals should be attributable to the laws, programs, and resources that EPA administers, but
broad enough to encourage substantial roles by other parties. Third, the goals should be
measurable, using available data and systems, although data quality and availability should not
entirely drive the choice of goals. These three criteria led to goals that:
Paralleled EPA's organizational structure in many cases, reflecting a combination of
environmental media (air and water) and programs (waste management).
Emphasized progress toward compliance with health- and ecological-based environmental
standards established in EPA regulations (for example, attainment of air quality standards).
Focused mainly on EPA's role and statutory responsibilities rather than on a broader scope of
federal environmental responsibilities.
Mindful that GPRA directed agencies to develop strategic goals that reflect the agencies'
statutory authorities, EPA developed goals that accurately reflect its role in setting federal,
health- and ecological-based environmental standards and implementing and enforcing major
federal environmental laws. However, these goals have fallen short of key partner and
stakeholder expectations in several ways, as follows.
Cross-Media Implementation
In 1996, EPA chose a 10-goal structure composed of 7 goals derived from the National
Environmental Goals Project and 3 additional non-environmental goals. This goal structure
reflected existing Agency programs and structure and required less change to linkages among
existing planning, budget, and accountability systems. In doing so, the Agency chose not to
adopt a broader, more multi-media oriented goal structure proposed later in the process by EPA
Region 5. According to EPA decision documents, this approach would have better supported
place-based and multi-media initiatives. EPA placed this more multi-media orientation in the
Agency's mission statement rather than in the specific goals and objectives used to track EPA's
efforts. EPA regions and the states have commented that EPA's 10-goal structure can restrict
cross-media implementation, often the preferred strategy for addressing local environmental
problems.
Non-Environmental Goals
EPA expanded its goal structure by adding three non-environmental goals (Sound Science,
Enforcement, and Effective Management) to provide an explicit, comprehensive system of
budget accountability under GPRA for each of EPA's major program offices. EPA stakeholders,
-------
however, have expressed concern that the non-environmental goals were process rather than
outcome oriented. In particular, states objected to EPA's enforcement goal as inconsistent with
GPRA's outcome orientation. States have consistently commented that enforcement is a key tool
supporting federal and state environmental efforts in achieving environmental goals, but should
not in itself be considered an environmental goal.
Expectations Beyond EPA's Authority
To accomplish its goals and objectives, EPA works in partnership with many other federal
agencies. This includes agencies on whom EPA relies as co-implementors of federal
environmental programs (such as the Army Corp of Engineers) and as members of the regulated
community (such as the Departments of Energy and Defense). EPA also works with federal
agencies involved with public health and ecological protection (such as the Public Health Service
and Department of the Interior).
EPA's goals are generally consistent with GPRA's requirements limiting agency goal
development to areas of direct statutory responsibility. However, as stakeholders noted in the
National Environmental Goals Project, the ultimate aims of environmental laws i.e., healthy
people and ecosystems and the federal partnerships necessary to achieve those larger goals, are
beyond EPA's scope of authority or responsibility under GPRA. If EPA were to attempt to meet
these larger stakeholder expectations in the future, it would be necessary for EPA to seek the
support of the Administration and Congress for developing broader goals in partnership with ^
other federal agencies.
Recommendation .
EPA has laid a solid foundation for future improvements with its goal development process.
There is a substantial record of stakeholder and partner opinion and preference. Stakeholders are
in substantial agreement over the importance of EPA goals for setting Agency direction and .
strengthening federal and state partnerships. In order for the Agency to assume a greater
leadership role through the future development of its GPRA goals, we recommend that EPA:
1 -1. Move expeditiously to amend EPA's goals and the processes for implementing them, as
necessary, to focus on environmental outcomes and encourage cross-media
implementation. EPA should also seek the support of the Office of Management and
Budget and Congress for a federal government process to develop environmental goals that
promote the ultimate human health and ecological aims of environmental laws, and the
federal partnerships necessary to achieve them.
MHadqUarters
a"
o
Pennsylvania Avenue NW
Washington DC 20460
-------
2. Priorities
Does EPA have a system for setting
priorities to best achieve its environmental
mission?
While EPA sets multi-year. Agency level planning and budgeting priorities, there is no readily
visible system or process used to identify them. Also, while EPA does consult informally with
stakeholders on planning and budgeting priorities, there is not a formal opportunity for comment
until submission of the budget to Congress.1 The priorities do not undergo any formal
examination of options, costs, arid benefits. As a consequence, EPA's multi-year priorities have
been shaped more by law and public perception than by science and cost.
System for Setting Priorities
Because resources are limited, EPA must set priorities among competing environmental
problems, programs, and issues. EPA's media program offices set and implement priorities
among the many objectives in EPA's strategic plan by setting 5-year performance goals and
allocating funds to them annually. To set priorities, program offices consider information on
legal authority, cost, environmental risk, and public perceptions. However, we found that
program offices do not document or explain how these factors are used in decision making.
In discussions with Agency staff, we
found that the specific statutory
requirements of EPA's laws, as well as
public perceptions, have shaped EPA
priorities more than science and cost
information. EPA's senior management
has not established risk based decision-
making criteria to set priorities for the
Agency. Although environmental risk
assessment is used by EPA to set specific
health-based standards, it is not usually
used for setting broad Agency and
programmatic priorities.
Historically, EPA's priorities have been shaped by
a variety of factors:
Environmental laws
V
Public perceptions of ^
environmental problems >v
X
Science/risk information**
Economic costs*'
EPA Priorities
Also, while EPA estimates the compliance costs of specific rules to assess cost-effectiveness,
there is no formal process for considering cost or cost effectiveness in Agency priority-setting
decisions. Thus, there is little assurance that EPA is allocating its limited resources to those
problems which pose the greatest environmental risks and opportunities for risk reduction.
1 OMB Circular A-l 1 Section 36, Communications with Congress and the Public and Clearance
Requirements, states, in part, that the nature and amounts of the President's decisions and the underlying materials
are confidential and are not to be released outside the Agency until the budget is transmitted to Congress.
-------
Consistency of Priorities
Each year, EPA identifies the Administrator's planning and budgeting priorities in budget
formulation guidance to EPA program offices. We identified 22 distinct Administrator priorities
in a review of Congressional budget summaries for fiscal years 1996 through 2000, covering
EPA's experience prior to and in the first year of GPRA implementation. Of these 22 priorities,
!
Administrator Priorities
Global Climate Change
Strengthen Slate and Tribal Partnerships
Improve Environmental Information
Brownfidds
Achieve Progress on Clean Air Programs
Protect Children's Health
Reinvent Environmental Regulation
Safe Drinking Water
Toxic Waste Sites
Clean Water
Sound Science
Livable Communities
Toxic Pollutants
Community- Based Environmental Protection
Use Advanced Technologies in Environmental Management
Eliminating Barriers to Regulatory and Permitting Process
. ~ ~" '
Implement Resource Conservation and Recover) Act Reinvention
Balanced Compliance and Enforcement
Implement Performance Partnership Agreements
Prated Human Health by Reducing Environmental Threats
Achieving Savings Through Streamlining efforts
Establish Working Capital Fund
Budget Years in Which Appear MllmhM. , .
1996
iiiS*
. '._.-,, .
1997 1998 1999 2000 Budgets
;>:;: ; -,5:; > T''-Ki;>..,,l;,Isft^-;: s
'"IN"?-- 'VN'4:':f*';:;';"|;^5\":j . «
."':.l>.'j ^'">;:!-v"V>;'';-j *
^'/.j t.*r.$-~J:*""~ j
HSivj'i gi-^4*3>' "": j
I.I; ':/>?- '.;>:\iA. ::..'". 3 3
r''. /". "
jY .."/ _
5,. - :f s :
,,..
'' "?*$
i .;-, ':
i ;:;,;> ,;,^:;!,;"j , 3
;-'. *' -"-.-v 1 ^ 3
j P;-- ." .-T,-. --:^.-1 3
i I?- .*! ' ^ . ^ "1 3
if; . ;.J. -;',.;»:.{ i 3
i X '! l.^;.jJl . .. . z
"' :-'* ^....^'f; 1 I 2
L."'t' J i 2
jv'^^J j ; i
f.'..--i'..v3 : ! i
|r%>"'j I l
l-?c '.;.! 1 i
\ iiji * ~" ' '* ^ '
: 1 i
! 1
only 2 appeared in all 5 years, and 11 appeared only once or twice:
The multitude of national priorities is a source of concern and confusion to EPA's regions and
the states, as noted in a recent National Academy of Public Administration report:
EPA's management system is stymied by a wide variety of priorities imposed by Congress and
the administration, and by itself. EPA staff are bewildered by the number of conflicting
priorities, statutory deadlines, court-imposed requirements and public participation needs.
States have challenged EPA to limit its priorities and devise a more flexible and open priority
-------
setting approach that accounts for state priorities.
Recommendation
By making EPA's priority setting clearer, the Agency can increase involvement of both internal
and external stakeholders. A more deliberate process for considering state priorities may begin
to address the criticism of EPA's national strategic planning process. In general, although risk
and cost information is imperfect, it is already used by some EPA program offices to set
priorities. Therefore, we recommend that EPA:
2-1. Define and document a national, multi-year priority setting process, and ensure that the
process fully and visibly uses available risk and cost information in setting priorities.
-------
11 :;<5#riii«»«ii£S^ ^as ^^ developed results-oriented
" « *Jr ? ?*'!! strategies that clearly link annual plans
! S J and budgets to accomplishments?
EPA has taken many actions to comply with GPRA's strategic planning requirements, although
it faces significant challenges in developing results-oriented strategies that clearly link annual
plans and budgets to accomplishment of long term outcomes. EPA has provided a GPRA
framework, guidance, and training, among other actions. However, EPA needs to better describe
how, when, and at what cost agency goals are to be achieved; the behavioral changes needed to
accomplish these goals; and the links between current activities and long term outcome goals.
Agency Progress in Strategic Planning
Since GPRA's passage, EPA has undertaken many actions
to comply with the Act's strategic planning requirements
and enhance the Agency's ability to manage for results. A
critical early step was the establishment of a framework
within the Office of the Chief Financial Officer (OCFO)
designed to align planning, budgeting, analysis, and
accountability in an integrated system. This framework is
designed to link EPA's annual planning and budgeting
decisions to EPA goals.
In addition, OCFO has: (1) provided guidance on the
planning, budgeting, analysis, and accountability system;
(2) facilitated workgroups to help implement the new
system; (3) issued guidance on how to link annual
performance goals to long term outcomes; and (4) assisted
national program managers in developing and delivering
GPRA training to selected staff.
Framework for GPRA
Implementation at EPA
Goals
Objectives
Subobjectives
I
;t
Strategies
Annual
Performance Goals
Annual Planning
And Budgeting
Another important step involved creating categories to aid program planners in understanding
where their environmental program goals and activities fall within a results-oriented hierarchy.
Known as the Hierarchy of Performance Indicators, program managers are now required to use
this environmentally-based categorization tool when developing their annual performance goals
and corresponding performance measures. According to OCFO guidance, this tool should help
the Agency move away from activity-based outputs to results-based outcomes. The most recent
categorizations from OCFO are shown in the following table.
-------
Modified Hierarchy of Performance Indicators
Level
1A
1B
1C
2
3
4
5
6
:R
' Measure of Progress
Customer Satisfaction - _ ' «
Service Delivery
Any EPA, state/tribal, other govemmentaK multi-governmental,
activities, tools, standards
and/or multinational regulations.
Actions/Responses by regulated community "
Changes in, or prevention of, volume, mass, and/or toxicity of discharges, loadings, or emissions
Ambient conditions
* .
Exposure or body burden/uptake
Risk or impacts to ecology, health, and/or welfare
Research and development
Agency Challenges in Strategic Planning
Although the actions taken by EPA to comply with GPRA's strategic planning requirements
have helped managers improve their focus on results, the Agency still faces a number of
challenges. These include: (1) describing in sufficient detail for external stakeholders how,
when, and at what cost goals are to be achieved; (2) ensuring that strategies clearly identify the
behavioral changes required to achieve the results-oriented outcomes desired; and (3) clearly
linking current activities to long term outcome goals.
Describing Plans in Sufficient Detail
Our analysis of the key components of five key GPRA strategic planning documents2 for three
EPA goals (air, water, and waste management) indicated that these strategies do not provide
sufficient detail to allow meaningful evaluation and input by interested stakeholders for the out
years. In each instance, the strategies were vague in their explanation of the actions that state,
local, and tribal agencies needed or would be expected to take to achieve'the program's outcome
goals. Congress anticipated that it may take several planning cycles to refine the strategic
planning process and that strategic plans would be continually improved as various planning
cycles occurred. According to the General Accounting Office's (GAO's) 1997 GPRA Guide,3
external reviewers of agency strategies should be able to understand and evaluate the linkages
between inputs, outputs, and outcomes and reasonably see which activities lead to what
intermediate and, eventually, long term outcomes. Vagueness in this critical aspect of the
" The documents are (1) FY 1999 Annual Performance Report - EPA's March 2000 GPRA submission to Congress (Version ID
3830 enacted in President's Budget, Version 1); (2) EPA's September 2000 Strategic Plan (EPA/190-R-00-002); (3) EPA's September 1991
Strategic Plan (EPA/190-R-97-002); (4) EPA's FY 2001 Annual Plan (EPA/205-R-00-002); and (5) EPA's FY 2001 Congressional
Justification (CJ), also known as the Budget Justification.
J Agencies' Strategic Plans Under GPRA: Key Questions to Facilitate Congressional Review, GAO/GGD-10.1.16, May 1997.
10
-------
agency's GPRA planning was previously noted by the House Science Committee in 1997 when
it examined EPA's first plan. The committee noted that
The EPA draft plan acknowledges the States as a significant partner in achieving
environmental goals. Yet beyond this acknowledgment, the State role (as well as
that of the Indian tribes) is unaddressed in the draft strategic plan.
Desire for greater understanding of the details about how EPA plans to achieve its GPRA goals
continues to be expressed by the public, members of Congress, GAO, and others. EPA officials
for air, water, and waste management recognize that their respective programs need to improve
their descriptions of how goals are to be achieved, when, and at what cost, particularly for
Years 2 through 5 of their strategies. They also pointed out that, in some instances, they have
developed more detailed strategies which could be cost effectively linked using the World Wide
Web. For example, air program officials noted that they developed a fairly extensive urban air
toxics strategy in 1999, setting forth the agency's strategic thinking 10 years into the future.
Similarly, water program officials noted that in 1998 they developed a clean water action plan
describing over 100 actions that EPA, Agriculture, and other federal agencies planned to take to
improve the Nation's surface water over a 7-year period. Lastly, EPA officials suggested that the
existing Hierarchy of Performance Indicators could be improved by establishing a category for
state, local, and tribal partners separate from EPA programs.
Ensuring Clear Identification of Behavioral Changes
A second challenge involves ensuring that strategies clearly identify the actions and behavioral
changes that the regulated community needs or would be expected to make in order for the
Agency to achieve its outcome goals. Our analysis of .the GPRA strategic planning documents
indicates that the air and water program strategies do not provide sufficient details about the
expectations for the regulated community. The strategies were vague in their explanation of the
actions that the regulated community would need to take to achieve the program's outcome
goals. The following table shows how these behavioral changes are a critical step in the logic
matrix for accomplishing EPA's outcome goals. .
11
-------
Logic Matrix for Accomplishing Goals
Note a
Noteb
Notee
-I INPUTS ft OUTPUTS
OUTCOMES '
;,Level 1
I Level2 [Level3
Level 4
Rewurcas
;IPA
Actions and
Actions by
Sane,
Local,
Tffcalerxf
other. ..'
! Rsoutatory
Agencies
Actions and
Behivioral
Changes "
bythe
:R«guteted
Community
Reduced-''
Amounts' <
; end/of *
ToxicSyo?
emissions
Improved ,. ;
ambiont
mania! : .
coraJKiocs .
Changes in
GGfluftlQfllS
Siwxlards
Levels I Level6
Reduced
exposure'
orbody
burden ,
Human ares
ecological
r Progress toward "\
'" long-term outcomes /"
__il_V__ ~_S
Hole a: Simple logic model categorizations.
Note b: OCFO Modified Hierarchy or Indicators
Note c: EP/VOIG expanded hierarchy or indicators
According to representatives of the national programs for air and water, using logic model .
concepts to perform a matrix analysis similar to that above would be helpful in understanding the
critical linkages in managing for results. They noted that it is'difficult to link behavioral
expectations in their media programs when the majority of activities and funding for the needed
actions are contained in a separate Agency goal. Although a critical link in the GPRA process,
they explained that the existing Agency goal structure, where compliance and enforcement is a
separate goal (Goal 9) generally funded and carried out separately from their media programs,
sometimes hinders their efforts to clearly identify the actions and behavioral changes that the
regulated community needs or would be expected to make in order for the Agency to achieve its
outcome goals. Representatives of both programs noted that EPA's decision to separate
compliance and enforcement goals from media program goals reflected extensive deliberations
and recognized that there would be tradeoffs; however, they also said it may be appropriate to
evaluate these-tradeoffs again.
Linking Current Activities to Long Term Outcome Goals
A third challenge the Agency faces involves clearly linking current activities to long term
outcome goals. A vague statement of actions beyond the near term makes it difficult for external
groups to see clear linkages, although the agency staff may be well aware of these linkages.
Additionally, the Agency's overall long term goal - - true human and ecological health - - is
shared with many other federal and non-federal entities, difficult and costly to measure directly,
12
-------
and hard to explicitly link to some near term agency activities. As such, strategies often rely
heavily on the fulfillment of largely EPA-controIled surrogate measures of human and ecological
health, such as a reduction in the emissions of regulated pollutants or a reduction in the number
of violations reported annually. The implicit assumption is that these reductions or changes will
eventually lead to true human and ecological health.
EPA's strategies recognize the need to rely on others for long term goal achievement. However,
EPA needs to provide more explicit descriptions of the partnering activities, roles, and
responsibilities for outcomes by these non-EPA groups. This would better enable external
reviewers to both understand and evaluate the linkages between inputs, outputs, and outcomes,
and accomplish GPRA's purpose of improving the public's confidence in the ability of federal
agencies to manage for results.
Recommendations
We believe EPA has an opportunity to improve its GPRA strategic plan and simultaneously
improve its relationship with state, local, and tribal agencies, as well as improve the public's
confidence in the Agency. As such, we recommend that EPA:
3-1. Link other, more detailed, planning documents to its GPRA strategic plan by providing
World Wide Web citations (links), thus providing external stakeholders with easy access
to more in-depth information on how EPA plans on achieving long term goals.
3-2. Revise the existing Hierarchy of Performance Indicators to clearly establish a place for
state, local, and tribal agencies' expected roles in achieving environmental goals and
proactively involve these partners in GPRA strategic planning for the next round of
strategic plans, due in 2003.
3-3. Work across agency goals to develop clear expectations for the behavioral changes EPA
expects industry and the regulated public to make if envisioned goals are to be achieved.
3-4. Work with other federal agencies that have human health and ecology goals to establish
long-term outcome-oriented goals that reflect the coordinated efforts of multiple federal
agencies.
f,H,eadquartere Library
1onnDMa" code 3201
1200 Pennsylvania Avenue NW
Washington-DC 20460
13
-------
4. Measurement
Does EPA have systems to
measure, evaluate, and report on
environmental results?
EPA has taken steps to put systems in place to measure and report on progress under GPRA, but
continues to have difficulty developing and obtaining the information necessary for measuring,
evaluating, and reporting on long-term environmental outcomes. Since GPRA's enactment, EPA
has aligned its management systems to help support implementation of GPRA, by establishing:
An Office of the Chief Financial Officer, to work with program offices to develop and track
Agency-wide environmental measures.
An Office of Environmental Information, to centralize responsibility for improving
interagency data sharing, as well as improve the accuracy, reliability, and scientific basis of
environmental information.
An Evaluation Network, to stimulate and promote evaluation as a valuable tool for EPA
program managers.
EPA has been recognized for its effort to align its budgeting, planning, and accounting systems
in an attempt to track and report on the resources used in achieving its GPRA objectives.
Nonetheless, EPA continues to face challenges in implementing GPRA. EPA's performance
information tends to focus on program activities (outputs) rather than results (outcomes), due to
the difficulty in obtaining and measuring environmental results. Further, the majority of EPA's
information is collected by states, and the inconsistencies in that information makes its use for
measuring, evaluating, and reporting difficult.
Measuring and Reporting on Results
EPA faces a number of challenges in attempting to
measure and report on environmental results. A
majority of EPA's annual performance goals and
measures focus on program activities or outputs
(number of enforcement actions, number of permits
issued, etc.). Despite the vast array of data in EPA's
information systems, GAO, the states, regulated
entities, and EPA itself have pointed out that the
Agency does not have much of the information it
needs pertaining to environmental conditions and
trends and the potential human health risks of various
pollutants. This makes it difficult to report on the
environmental results of EPA's activities. Further,
Availability of Environmental
Results Data
Activity/Output
Hinunand
Ecological H«i!»
Oulcomet
I V
14
-------
environmental programs may take several years to show results. Thus, measuring and reporting
on end results may also take years. In addition, even with better measures and data, linking
environmental programs and activities to outcomes is complicated by a myriad of external
factors, including weather, international environmental issues, and economic activity, many of
which are outside of EPA's control. -
Further, while EPA does not currently receive all state information, a majority of the information
that EPA uses for measuring and reporting is collected by states. According to the
Environmental Council of States, some 94 percent of all environmental quality data in national
databases originates with the states. In previous reports, the OIG, as well as others, have found
data for individual states to be inconsistent, and data among various states to be incompatible.
States are encouraging EPA to allow more flexibility in what data is collected and how it is
collected. According to the Environmental Council of States, the states would rather concentrate
on setting broad indicators of environmental improvement and state capacity rather than on
traditional statistical and financial measures. This concern is echoed by the regulated
community, which is urging EPA to focus on defining, measuring, and rewarding environmental
results and reorienting core regulatory functions so they are driven primarily by performance
goals and not activity measures. While the EPA's Office of Chief Financial Officer continues to
work with program offices to develop measures that demonstrate environmental.outcomes, it is
proving to be a difficult and complex task.
Evaluating and Reporting on Results
There is an important distinction between performance measurement and program evaluation.
Measurement is the ongoing monitoring and reporting of program accomplishments and seeks to
answer the question, "what was achieved?" Program evaluation attempts to determine the
manner and extent to which programs achieve intended objectives, usually through objective
measurement and systematic analysis. The intent is to analyze why results were achieved, so that
further action can be taken based on lessons learned. Thus, evaluation is aimed at providing the
link between actions taken and results achieved.
According to GAO, in some programs, outcomes are not quickly achieved or readily observable,
or their relationship to the program is uncertain. In such cases, program evaluation is needed,
along with performance measurement, to examine the extent to which a program is achieving its
objectives.4 EPA has been without a centralized evaluation function since 1995, when the
Program Evaluation component of the Office of Policy, Planning and Evaluation was disbanded.
More recently, EPA established a Program Evaluation-Network to help stimulate interest in
evaluation and increase the Agency's capacity to conduct evaluations. The Office of Policy,
Economics and Innovation created a division to evaluate EPA's many reinvention initiatives and
help support and promote evaluation activities throughout the Agency. OIG has also established
an Office of Program Evaluation to evaluate EPA programs.
4 Program Evaluation: Studies Helped Agencies Measure or Explain Program Performance
(GAO/GGD-00-204)
15
-------
While the Office of Chief Financial Officer's Office of Planning, Analysis and Accountability
has an analysis component, the staff in this office do not see their role as conducting evaluations
for EPA program offices. Rather, they view their role in evaluation as facilitating and helping
EPA programs develop capacity through mechanisms such as the evaluation network. EPA
program offices are beginning to focus some attention on conducting evaluations, but the
resources they are devoting are limited and the distribution of those resources uneven from
program to program. Additionally, EPA has not established policy or guidance on how program
evaluation results are to be used and tracked by management. Without such controls, there is
little assurance that evaluative results will be used.
Recommendations
For EPA to demonstrate and report on the outcomes and impact of environmental programs, a
concerted effort will be needed between EPA, states, and the regulated community to develop
outcome-oriented performance information and environmental indicators. Where such
information does not exist or will take time to develop, program evaluation can help bridge the
gap and examine the extent to which programs are achieving their objectives. Consequently, we
recommend that EPA:
4-1. Work with states and the regulated community to develop an information strategy that
includes:
Clearly defined human health and ecological outcomes, as well as output measures.
Expanded use of environmental indicators and performance-based management.
Clear and consistent data definitions and standards for improving data quality.
Identification of the resources necessary to obtain, compile, store, and share data.
4-2. Encourage the use of evaluation staff as a resource for program managers and develop
guidance for how evaluative results should be used and tracked.
16
-------
Does EPA have the human resource
systems needed to manage for results?
EPA recognized the importance of human capital as a key Agency priority in its Strategic Plan,
and has recently developed and begun implementing a separate Strategy for.Human Capital.
However, in its Strategic Plan, EPA did not specifically address the human capital needs of any
of its strategic goals. A prerequisite for EPA's successful implementation of GPRA is the
application of the right skills and competencies within a results-oriented culture. EPA needs to
better integrate human capital into strategic plans; more effectively develop competencies in
management, science, and technical skills; and better align individual accountability with
Agency goals and results.
Human Capital.Emphasized
GPRA expresses the importance of planning human capital needs to attain Agency goals.
Several independent evaluations by GAO, National Academy of Public Administration,
Environmental Council of States, National Research Council, and EPA's OIG have focused on
aspects of EPA's human capital management. These evaluations addressed the need for
developing and aligning scientific and management competencies, as well as individual
performance expectations and accountability, with strategic goals. For example:
GAO stated that, "successful human capital management strategies are the key to maintaining a
highly skilled, energized, and empowered workforce that is focused on results'."
GAO also stated. "EPA is not able to identify the size of its workforce and the competencies that
need to be deployed among its organizational components to effectively carry out its strategic goals
and objectives."
The National Research Council stated that, "to achieve scientific and technical excellence, EPA must
attract, retain, and properly support first rate dedicated professional staff."
The OIG recently reported in Superfund that the Headquarters program office and several regions
did not identify the quality assurance training needs, and in some regions training was not always
provided. Also, audits have repeatedly noted a need .to better train managers in their
administration and oversight of EPA's assistance agreements. Additionally, we found EPA
employees in the hazardous waste program needed more rigorous training to calculate proposed
penalties against violating facilities, and that the National Environmental Performance
Partnership System (NEPPS) has not been well-integrated into EPA due to lack of, among other
things, training and guidance. Human capital concerns will intensify in the coming years as 52%
of Senior Executive Service, 47% of scientific staff, and 25% of EPA's total workforce will be
eligible for retirement within 5 years, while competition for scientific, technical, and managerial
talent will likely increase.
17
U'S- EPf Headquarters Library
Mail code 3201
S/Peu nsylvania Ayenue NW
Washington DC 20460
-------
Human Resource Elements Essential
Effective implementation of performance-based management as envisioned by GPRA is
dependent on senior management's willingness and ability to strategically manage all of the
Agency's resources, including human capital. Specifically, as noted by GAO, this requires
aligning and developing day-to-day workforce competencies and expectations with strategic and
program plans and results at all levels for efficient and effective application of limited human
resources, and to promote a results-oriented culture. Human resource elements essential for
successfully implementing GPRA and attaining EPA's goals are:
Integrating Human Capital into Strategic Plans
Developing Competencies in Management, Science, and Technical Skills
Aligning Individual Accountability with Agency Goals and Results
Our review of these elements noted the following future challenges EPA faces in attaining its
goal of having the human resource systems needed to manage for results.
Integrating Human Capital into Strategic Plans
The Agency recognizes that one of its biggest challenges over the next several years is the
development and implementation of a workforce planning system that focuses attention on the
identification of needed skills and competencies, and the means of addressing them. EPA
developed a Workforce Development Study to discuss competencies needed to: meet the current
Agency mission; identify possible future missions of the Agency; and suggest new competencies
for future scenarios. EPA has broadly recognized the importance of human capital as a key
Agency priority in its September 2000 Strategic Plan, but has not specifically addressed human
capital needs in its goals. Although neither the Strategic Plan nor the Workforce Study address
specific human capital issues or make commitments to specific human capital actions, EPA is
developing a more specific workforce development plan.
EPA needs to follow through on its recently developed Strategy for Human Capital by making a
consistent commitment to evaluate the number of staff and competencies needed, and how those
employees should be deployed among its program areas and in various locations to meet its
strategic goals. GAO recently reported that while EPA has implemented several initiatives
during the past decade to better understand the demands facing its workforce, these initiatives
have not received the resources and senior management commitment to bring them to fruition.
Therefore, EPA needs to implement a strong management approach to training and development,
and organize its efforts around principles agreed upon by senior Agency management. The need
for developing, integrating, and aligning human capital planning with the strategic and annual
plans will be magnified as: EPA's role changes; the need for specific sophisticated scientific,
management, social, and technical skilis increases; the current workforce ages; and competition
to recruit and retain highly qualified staff intensifies.
18
-------
Developing Competencies in Management, Science, and Technical Skills
One of EPA's long-standing human capital'challenges is attaining and developing the needed
staff competencies in management, scientific, and technical positions. EPA has not assessed the
gap between (1) critical skills and competencies needed to achieve its goals and objectives and
(2) existing staff skills and competencies. Furthermore, appropriate training is essential for
management to lead and promote a results-oriented culture in EPA and for scientific and
technical staff to achieve and maintain excellence and credibility among their peers. However,
National Research Council, GAO and OIG reviews found that EPA staff often lacked specific
training for technical, scientific, and financial program responsibilities. '
Aligning Individual Accountability with Agency Goals and Results
GPRA requires EPA to clearly establish results-oriented performance goals for which it will be
held accountable. Since Agency accountability is dependent upon the collective accountability
. of its staff, EPA must align individual employee'performance expectations with Agency goals.
This is necessary so that managers and staff understand the connection between their daily
activities and the Agency's progress in accomplishment of its goals. As described by EPA staff
at different levels and components, EPA has not consistently shifted the orientation of individual
performance expectations and accountability from process or activity completion to achieving or
contributing toward attainment of Agency goals. EPA also has not integrated a GPRA results-
based accountability framework and culture into staff day-to-day activities by regularly
incorporating results-based expectations into its staffing, performance management, and
development documents, such as vacancy announcements, position descriptions, performance
agreements, and individual development plans.
Recommendations
For EPA to achieve its strategic goals, it must focus on how to best invest in its people and link
their responsibilities and performance to Agency results. EPA is taking steps in the right
direction with its workforce development plans and strategies, but now must make decisive
commitments and investments to integrate human capital requirements into its Strategic Goals
and Annual Plans. In order for EPA to effectively implement GPRA, we recommend that EPA:
5-1. Identify needed scientific, technical, and management competencies to achieve specific
mission goals, and measure the extent to which employees possess these competencies.
5-2. Fund and implement training and development to address skill gaps.
5-3. Apply management, technical, and scientific competency criteria to employee
recruitment, selection, development, and performance accountability processes.
5-4. Align staff performance expectations and contributions with Agency goals.
19
-------
6.
Has EPA established an
accountability system to ensure
efficient achievement of its goals?
EPA has established an accountability system with the intent of managing resources efficiently,
with information to ensure the effective achievement of its goals, and has been recognized as a
leader in developing a goals-based budget aligned with its programmatic and operational outputs
and outcomes. However, this framework, while well intentioned, tends to perpetuate EPA's
"stove-piped" organization. Consequently, it does not address overlapping environmental issues,
as well as the different needs and priorities of EPA's partners - the states - who implement the
majority of EPA's programs. EPA's system of accountability for attainment of its goals do not
sufficiently link or align authority, responsibility, resources, measures, and outcomes.
Statutes Require Accountability
Congress' desire to hold agencies accountable for their performance and results was the
motivating force behind the Chief Financial Officers Act of 1990 (CFO) and GPRA. While the
CFO Act established the foundation for improving management and financial accountability,
GPRA created requirements for agencies to generate additional information Congressional and
executive branch decision makers need in considering measures to improve government and
reduce costs. The purpose of GPRA, as stated in the law, is to systematically hold federal
agencies "accountable for achieving program results." Additionally, the Reports Consolidation
Act of 2000 authorizes agencies to combine performance information with cost information, to
impose a new, more business-like framework for management and accountability for federal
agencies. Appropriate information on results and costs is necessary for accountability.
EPA Has Established an Accountability System
EPA has established an
accountability system tied to its
goals, as shown in the accompanying
chart. The Agency has taken a goals-
based budget that is aligned with its
programmatic and operational
outputs and outcomes as a means of
accountability and to accomplish its
goals. EPA has been recognized as a
leader in developing this innovative
approach.
EPA's Accountability System
Annual Performance
Plan and Budget
Memorandaof Agrtemtm £.
Grant
WorkplansrPerfonnance
Partnership Agreements |
Annual Performance
Reportf Mid-Year Deputy
Administrator Meetings
Regional Mid-Year and
Annual Reporting
Accomplishment Reports
20
-------
Essential elements for an accountability system that will ensure efficient achievement of EPA
goals are:
Alignment of EPA Goals with Stakeholder Multimedia Priorities
Agreement on Roles, Responsibilities, and Expectations
Comparability and Standardization of Results-Oriented Measures and Data
Linkage of Cost Accounting Information with Activities used for Decision Making, and
Results
Our review of these elements noted the following challenges EPA faces in attaining its goal of
having an effective accountability system.
Alignment of EPA Goals with Stakeholder Multimedia Priorities
EPA establishes its long-term and annual goals and develops its budget by goal. Accountability
for progress and the use of resources by EPA goal is not consistent with the flexibility necessary
for multi-media activity, investment, and priorities that EPA regional offices need in order to
address the interests of state partners. For example, EPA regional officials told us that EPA's
goal structure tends to promote a stove-piped organization and that it is not conducive to cross-
media work, such as Environmental Justice, Brownfields, and Children's Health. EPA needs to
develop a process to accurately capture state priorities, and where agreed upon, adjust budgets
and accountability to reflect those priorities.
Agreement on Roles, Responsibilities, and Expectations
EPA depends heavily, on states to fund and implement national programs as well as provide most
of the environmental data. The charts below demonstrate the trend in environmental work done
by the states in terms.of percentage of the programs that have been delegated to the states, and
the relative percentage of EPA's investment in environmental protection compared with that of
the states. Also, according to the Environmental Council of States, at least 80 percent of all
environmental enforcement actions are taken by the states.
Percent of Environmental Programs
Delegated to the States
FY 1996 Environmental Investment
By States & U.S. EPA
$ in Billions
States
- $12.5
65.8%
Source: Environmental Council of States (ECOS)
Total: $19.0
21
-------
EPA and states have not yet agreed on how states should be provided with flexibility, along with
accountability, for environmental results. Relations between EPA and states have been strained
due to disagreements over:
Respective roles and the extent of federal oversight
Priorities and budgets
Results-oriented performance measures, milestones, and data
Comparability and Standardization of Results-Oriented Measures and Data
There is limited standardization and comparability of data between states. As already noted, the
Environmental Council of States estimates that about 94 percent of the national data on
monitoring of environmental quality is collected by the states, and lack of compatibility of data
makes it difficult to aggregate data for trending, comparing, and evaluating performance and
progress.
Linkage of Cost Accounting Information with Activities and Results
Since Fiscal Year 1999, EPA has been able to link its budget and costs to goals, objectives, and
sub-objectives, but not often to activities and outcomes. For example, EPA's FY 2000 Annual
Report does not provide cost information associated with its annual performance goals and
measures. According to the Office of the Chief Financial Officer, at least one EPA region is
piloting a cost accounting and performance reporting tool, and EPA has also adopted cost
accounting practices in the Superfund program, Oil Spill programs, and in the Working Capital
Fund. EPA has not consistently decided what level of cost accounting is needed for decision
making in all programs and has not defined an approach for using cost information. EPA's cost
accounting system generally does not track costs below the Sub-Objective (i.e., 1-5 year) level,
nor does it track across goals as demonstrated in the diagram below. The Agency recognizes the
need for cost accounting for key activities and is currently developing cost accounting tools to
better respond to Agency needs. To obtain this information, EPA should identify the key
activities for which it needs to accumulate costs and develop a means for collecting and using this
information for results based management, program execution, and evaluation.
Hierarchy of EPA Resource Allocation
A cost accounting system linking activity costs with performance,
would enable EPA to:
/ Identify Return of Investment
v1*" Compare investment options and trade-offs
/ Account for many regional and state multi-media activities
linked to multiple sub-objectives
/ Promote and recognize greater operational efficiencies
/ Make optimal budgeting and planning decisions
Goal
Objective
Sub-Objective
""" ' . ' : . «" :.
I
"4
Strategy/Activity
Annual Performance
Goal
Annual Planning &
Budgeting
22
-------
Recommendations
EPA has many federal and state partners and stakeholders that collectively have a variety of
needs, levels of involvement, and interdependence in contributing to EPA's national goals. EPA
can improve its accountability by seriously considering state and stakeholder priorities and
responsibilities in its planning, budgeting, and reporting systems, and aligning its goals, activities
and resources with performance measures and results. The Agency has the capacity to link its
budgeting, planning, and accountability information at a high level and must follow through on
the actions it has started to improve performance and cost data. To strengthen accountability
processes, we recommend that EPA:
6-1. In setting expectations:
Define roles and responsibilities for EPA and its partners;
Agree on priorities, strategies and results; and
Develop and use better environmental performance measures and data.
6-2. In the short-term, integrate existing cost and performance data, through Agency-wide cost
accounting, to:
Identify return on investment, both at the activity and outcome level;
Compare different investment options for future investment choices; and
Promote and recognize greater efficiencies and opportunities for savings.
6-3. In the long-term, identify the key activities for which it needs to accumulate costs and
develop a means for collecting and using this information for budget formulation and
planning, program execution, and evaluation.
23
-------
24
-------
Exhibit 1
Scope and Methodology
Purpose of Review
To determine what progress EPA has made in using GPRA to manage for environmental results,
we conducted a review of GPRA's implementation activities. The intent was to inform EPA's
leadership and interested stakeholders about EPA's decisions that set the stage for GPRA
implementation; progress in using GPRA; challenges that EPA faces in implementing GPRA; and
opportunities for near and long-term improvements. .
Questions Posed .
To conduct our review, we posed questions related to six different critical aspects of GPRA:
1. Goals - Are EPA's goals consistent with its statutory authority and key partner and
stakeholder expectations?
2. Priorities - Does EPA have a system for setting priorities to best achieve its
environmental mission?
3. Strategies - Has EPA developed results-oriented strategies that clearly link annual plans
and budgets .to accomplishments?
4. Measurement - Does EPA have systems to measure, evaluate, and report on
environmental results?
5. People - Does EPA have the human resource systems needed to manage for results?
6. Accountability - Has EPA established an accountability system to ensure efficient
achievement of its goals?
Review Steps Taken
To answer these questions, we interviewed EPA Headquarters and regional officials, and also
discussed GPRA implementation with environmental officials in five states. In addition, we
reviewed previous reports, publications," and correspondence related to GPRA implementation
written by the:
General Accounting Office
Environmental Council of States
Senate Governmental Affairs Committee
House Oversight and Reform Committee
. . 0 _
U-S. EPA .Headquarters Library
Mai/ code 3201
1200 Pennsylvania Avenue NW
Washington DC 20460
-------
National Academy of Public Administration
* National Research Council
George Mason University's Mercatus Center
We also took into account information in other reports published by the EPA OIG.
We reviewed Agency-wide projects on core competencies and suggestions by states and the
regulated community for improving EPA's management of environmental programs. We
reviewed the GPRA framework for EPA's Offices of Water, Air and Radiation, and Solid Waste
and Emergency Response (Goals 1, 2 and 5). We reviewed publicly available documentation,
such as EPA's Strategic Plan, Congressional Budget Justification, Annual Performance Plan, and
Annual Performance Report; as well as comments from the public, states, and the regulated
community regarding these documents. This review was considered a special project and, as
such, was not conducted in accordance with Generally Accepted Government Auditing Standards.
26
------- |