OIG
V
"'l PRO**6
OFFICE OF INSPECTOR GENERAL
Catalyst for Improving the Environment
Audit Report
EPA Needs to More Actively
Promote State Self Assessment
of Environmental Programs
Report No. 2003-P-00004
December 27, 2002

-------

-------
s
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
OFFICE OF
INSPECTOR GENERAL
December 27, 2002
MEMORANDUM
SUBJECT: Report Number: 2003-P-00004
EPA Needs to More Actively Promote State Self Assessment
of Environmental Programs
FROM:
Michael A. Rickey, Director
Assistance Agreement Audits
TO:
Edward Krenik
Associate Administrator for Congressional and Intergovernmental Relations
We completed our preliminary research of the effectiveness of state self assessments. The
objective of the assignment was to answer the following questions:
•	Are states using self assessments to develop their performance partnership agreement and
performance partnership grant priorities and activities?
•	What have been the benefits and obstacles to conducting and using self assessments?
•	Have states that have based their performance partnership agreement and performance
partnership grant priorities and activities on a self assessment achieved environmental
results superior to those states that have not conducted such an assessment?
The purpose of preliminary research is to gather information to define the scope and
methodology for an audit. During preliminary research, we determined, as discussed below, that
the U.S. Environmental Protection Agency (EPA) has done little to develop and promote greater
reliance on state self assessment. We met with the Deputy Associate Administrator for
Intergovernmental Relations and discussed the results of our preliminary research. Based on our
work to date and the meeting, we decided to report on what we found and not to perform
additional audit work. This report addresses the first two objectives. Based on our conclusions
on the first two objective questions, we did not believe that there was sufficient information to
answer the third objective question. Details on the scope and methodology are in Appendix A.

-------
Background
In 1995, EPA and the states established the National Environmental Performance Partnership
System (NEPPS) to strengthen the management, efficiency, and effectiveness of the nation's
state and Federal environmental programs by improving the partnership between EPA and the
states. Self assessment is one of the principal components of NEPPS. Self assessment was
introduced as a new approach to program assessment, to share with the public information about
environmental goals, priorities, and prior year achievements. In the self assessment, states would
provide information identifying:
•	What the state sees as the key environmental problems, opportunities, and priorities facing
the state;
•	The recent performance of the state's programs based on available measures of program
success;
•	An analysis of current program weaknesses from the state's perspective;
•	An assessment of basic fiscal accountability, along with an identification of any areas
needing capacity building;
•	The state's proposed action plan for maintaining and improving its program performance,
and identifying specific action and approaches the state plans to take in the coming year; and
•	How well the state carried out the plan agreed to in the environmental performance
agreement.
While states are not required to participate in the NEPPS process, 40 Code of Federal
Regulations Part 35.115 requires regions and states to have a joint evaluation process for all
continuing environmental grants awarded after January 2001. The joint evaluation process
includes a discussion of existing and potential problem areas and suggestions for improvement,
including, where feasible, schedules for making improvements. The joint evaluation process
could provide EPA and the state with information similar to the self assessment that could be
used in setting priorities for each state.1
EPA and States Not Effectively Using Self Assessments
EPA and states had not widely adopted the self assessment concept. Many states were not
performing self assessments, their content varied, and they had little impact on environmental
performance agreements. This occurred because EPA had not taken a leadership role to define to
staff and states its expectations for self assessments. As a result, Federal resources may not be
directed at the state level to where they would have the most vital environmental impact.
Most states did not conduct self assessments. According to region officials, only 23 states
performed self assessments. Also, self assessments did not always address the elements
identified in the NEPPS policy. We reviewed self assessments for two states in Region 1 and
compared the contents of the documents to the elements prescribed in the NEPPS policy. Each
' As part of this audit, we did not evaluate whether regions and states were performing joint evaluations.
2

-------
state's self assessment varied from NEPPS policy. In one state, the self assessment addressed
five of the six elements. The other state's self assessment fully addressed two elements and
partially addressed two elements. The checkmarks in the table denote the elements included in
each state's self assessment.
Elements of a Self Assessment
State 1
State 2
Key problems and priorities
/
Limited
Recent performance
/
/
Program weaknesses
/

Assessment of fiscal accountability and capacity building


Plan accomplishment
/
/
Proposed action plan for improving areas
/
Limited
Self assessments did not include all of a state's programs. According to information provided by
the regions, 9 of 23 states did not include all programs in their self assessment. According to
officials in one state, all programs were not included because of the extensive coordination and
resources required to perform a comprehensive assessment.
According to state officials, the self assessment had little impact on the negotiation of the
environmental performance agreement and the allocation of Federal funds. State and regional
officials both stated that the negotiation process was dominated by EPA priorities. The NEPPS
policy states:
The self-assessment, in concert with EPA 's perspective on environmental conditions and
program performance, wouldform the basis for negotiating the Environmental
Performance Agreement.
Regional staff stated that the self assessment was of limited value to them. The self assessment
was used as a reference for state accomplishments and outputs, but was not very timely for
negotiation of the performance agreement. Further, some regional staff stated that the self
assessment will be of little value until EPA reaches a consensus on (a) what will be in the self
assessment, (b) how it should be conducted, and (c) how the results will be used.
Since the NEPPS policy was issued in 1995, EPA had done little to develop and promote greater
reliance on the self assessment. EPA had not issued any additional guidance or training on self
assessments. As a result, few regions and states were performing and effectively using self
assessments. EPA had not taken a leadership role in defining or re-defining the purpose of the
self assessment, what should be included, and how it should be used.
The NEPPS policy envisioned the self assessment as one of the tools for directing scarce public
resources toward improving environmental results and allowing states greater flexibility to
3

-------
achieve those results. By identifying and reporting on state priorities, performance, areas of
weakness, and fiscal accountability, the self assessment could provide valuable information on
where environmental resources are most needed in a state. Without a comprehensive self
assessment, Federal resources may not be directed to the state level where they will have the
most vital environmental impact.
Recommendations
We recommend that the Associate Administrator for Congressional and Intergovernmental
Relations:
1.	Clarify the purpose of self assessments and how they are to be used by EPA and states.
EPA needs to work with states to develop guidance and training on self assessments.
Questions to consider include:
What information should be included in the self assessment?
When should the self assessment be completed?
How should EPA combine or link self assessments with other reporting
requirements, such as the joint evaluation process required by 40 Code of Federal
Regulations Part 35.115?
How should EPA use the results of the self assessment to prioritize and allocate
Federal funds?
2.	Work with other EPA offices and states to gain their acceptance and involvement with the
self assessment concept.
Agency Response
In the Associate Administrator's November 20, 2002, response to our draft report (Appendix B),
he generally agreed with all of our findings and recommendations. The Associate Administrator
indicated that states that conducted self assessments considered the exercise worthwhile, but
expended considerable resources without EPA feedback. The Associate Administrator agreed
that a joint evaluation process similar to that required under 40 Code of Federal Regulations Part
35.115 would improve joint planning and priority setting, and would provide information similar
to the self assessment. EPA will continue discussions about advancing performance partnership
agreements with states. Based on the discussions, the self assessment process may be overtaken
by improved priority setting and joint evaluation processes.
Office of Inspector General (OIG) Comments
We agree with the Associate Administrator's response and that a joint evaluation process could
serve the same purpose as a self assessment process. At the exit conference with the Office of
Congressional and Intergovernmental Relations, we indicated that regardless of whether the
Agency chooses to use a joint evaluation or a self assessment, the Agency needs to clarify the
4

-------
process. Further, the Office of Congressional and Intergovernmental Relations should work with
EPA offices and states to gain acceptance of the process, as stated in the recommendations.
Action Required
In accordance with EPA Manual 2750, you are required to provide a written response to this
report within 90 calendar days of the date of this report. You should include a corrective actions
plan for agreed upon actions, including milestone dates. We have no objections to the further
release of this report to the public.
This audit report contains findings that describe the problems the OIG has identified and
corrective actions the OIG recommends. This audit report represents the opinion of the OIG and
the findings contained in this report do not necessarily represent the final EPA position. Final
determinations on matters in this audit report will be made by EPA managers in accordance with
established audit resolution procedures.
If you or your staff have any questions about this report, please contact me at (312) 886-3037, or
Ricardo Martinez at (212) 637-3045.
5

-------
6

-------
Appendix A
Scope and Methodology
In order to accomplish our objectives, we collected information from EPA's Office of
Congressional and Intergovernmental Relations, and from regions and states. Specifically:
•	We sent questionnaires to staff in EPA Region 1 and its six states regarding self assessment
use, benefits, and obstacles. For other than two states, we did not verify the information
provided in the questionnaires.
•	We interviewed NEPPS staff from the Office of Congressional and Intergovernmental
Relations, Regions 1 and 2, and the States of Massachusetts and New Jersey.
•	We obtained information on the number of states conducting self assessments through a
questionnaire to the NEPPS coordinators for all 10 EPA regions, and reviewed their
responses. We did not verify the information the regions provided.
•	We compared the content of the self assessments for Massachusetts and New Hampshire
with the NEPPS policy.
We conducted our fieldwork from August through October 2002. We performed our audit in
accordance with Government Auditing Standards, issued by the Comptroller General of the
United States. We limited our preliminary research to objective questions 1 and 2; and state
participation, benefits, and obstacles in conducting self assessments. Based on our preliminary
results and a meeting with the Office of Congressional and Intergovernmental Relations, we
decided to report on what we found and not to perform a full audit.
7

-------
8

-------
Appendix B
Page 1 of 7
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, DC 20460
November 20, 2002
OFFICE OF CONGRESSIONAL AND
INTERGOVERNMENTAL RELATIONS
MEMORANDUM
SUBJECT:
Draft Report - EPA Needs to More Actively Promote State Self-Assessment
of Environmental Programs
Assignment Number 2002-000877
Edward D. Krc^iLB~c0 £>¦
Associate Administrator for Congressional and Intergovernmental Relations
FROM:
TO:
Michael A. Rickey, Director
Assistance Agreement Audits
Thank you for the opportunity to comment on the report about your preliminary research
on the effectiveness of state self-assessments. Thank you also for your earlier meeting with Dona
DeLeon, Deputy Associate Administrator for Intergovernmental Relations, to discuss the results
of your research.
We appreciate the time you devoted to this important aspect of State/Federal relations and
generally agree with your findings. Since January 2002, the Office of Congressional and
Intergovernmental Relations (OCIR) has coordinated discussions between EPA senior managers
and state commissioners relative to how the performance partnership system is working. This
general evaluation and recommended next steps are in the attached draft, "The State-EPA
Performance Partnership for Cleaner Air, Purer Water and Better Protected Land." You will note
that on page 3 of the document the States conveyed that those that prepared self-assessments
invested considerable resources with no feedback from EPA; they believed the exercise was
worthwhile.
The paper suggests that identifying and reporting on state priorities, performance, areas of
weakness, and fiscal accountability could be accomplished through a joint assessment or as part
of a State-Regional Office joint strategic planning and priority-setting process. We agree with
your point that a joint evaluation process similar to that required under 40 Code of Federal
Regulations Part 35.115 would improve joint planning and priority-setting. We appreciate your
bringing this important requirement to our attention and we plan to formally include joint
evaluation in our work with the States. This joint evaluation process could include discussions
of existing and potential problem areas, roles and responsibilities of each level of government in
Internet Address (URL) • http://www.epa.gov
Recycled/Recyclable • Printed with Vegetable Oil Based Inks on Recycled Paper (Minimum 25% Postconsumer)
9

-------
Appendix B
Page 2 of 7
2
solving the problem and resources needed. We see this joint evaluation process providing EPA
and the State with information similar to the State self-assessment and it could be used in setting
priorities for each State.
EPA will continue discussions about advancing PPAs with the ECOS Strategic Planning
Committee. It is unclear at this time what role, if any, a State self-assessment will play. It very
well may be overtaken by an improved priority-setting process and a much needed joint-
evaluation process, as you point out in your review.
Should you or your staff have any questions about these comments, please contact the
Deputy Associate Administrator for Intergovernmental Relations, Dona DeLeon, at (202) 564-
7178 or Steve Wilson at (202) 564-3646.
Attachment
10

-------
Appendix B
Page 3 of 7
DRAFT
The State-EPA Performance Partnership for
	Cleaner Air, Purer Water and Better Protected Land	
For Discussion
at the
ECOS Partnership Meeting
April 23, 2002
4:15 to 6:15 PM
Co-chaired
by
ECOS Secretary-Treasurer Karen Studders, Commissioner, Minnesota Pollution
Control Agency
&
Dona DeLeon, EPA Deputy Associate Administrator for Intergovernmental
Relations
Successes: What Have We Accomplished ?
States and EPA agree that the May 17, 1995 Performance Partnership agreement put forth
a sound set of principles to maintain a core level of environmental protection for all
citizens. The principles guides a flexible process that adapts environmental goals to local
conditions in a way that builds trust between states and EPA and serves both of us as we
work together.
The National Environmental Performance Partnership System (NEPPS) greatly improved
trust and communication between EPA and State Environmental Agencies by
encouraging more frequent discussions between Regional Administrators and State
Environmental Commissioners and beginning to break down organizational and media-
program barriers at both the EPA regional and state offices.
One result of these improved communications was improved priority setting and planning
among state environmental agencies and EPA Regional Offices. This has been
demonstrated through: 1) improved internal priority planning by some state agencies; 2)
increased joint EPA Regional and state priority setting and planning efforts (i.e., EPA
Region V State's Environmental Performance Partnership Agreements); and 3)
1
11

-------
Appendix B
Page 4 of 7
DRAFT
development of self-assessments and strategic plans by some state agencies that improved
internal or joint priority setting and planning. In some instances these efforts led to
funding shifts and work sharing.
The improvement in priority setting and planning in turn better positioned EPA and the
states to focus national environmental protection efforts on results, creating a strong
management partnership. It created demand for better integration of environmental data
that yielded EPA's grant program to states, known as the "One Stop" grant program, the
subsequent state/EPA Information Management Working Group, and EPA's structural
reorganization creating the Office of Environmental Information. The states and EPA
are now building an internet based Information Exchange Network, which will increase
the speed of sharing data while driving down cost and improving efficiency, and
accuracy. Also, the EPA/State Regulatory Innovation Agreement developed in part from
these efforts, and for the first time, states and EPA agreed on performance measures to
track progress towards results.
Challenges: Impediments to Advancing Performance Partnerships
EPA's multiple priority setting and planning processes including performance
partnerships, development of national program guidance, and budgeting and
accountability requirements of GPRA developed too independently of each other and at
different times, and are not aligned in a way that fosters joint planning and priority setting
across media program lines.
Staff in both EPA and state environmental agencies have limited experience with a non-
traditional, collaborative approach to environmental problem solving. Also, formal
training for staff on how to manage programs to better focus on outcomes never occurred
and tools are limited. A strong media program perspective and loyalties still dominate
many aspects of state/EPA relationships and there are few incentives for state and federal
staff to take on new ways of doing business or to face new risks.
A Performance Partnership Agreement (PPA) is "in addition to" and many times conflicts
with delegation agreements, national program guidance or other documents dictating
some aspect of state -federal management of environmental programs.
Transaction costs for developing performance partnership agreements are believed to be
too high because of the perceived lack of flexibility in the process (# of hours spent
planning exceeds number of hours doing environmental work). The expected benefits of
a reduction of oversight and reporting were not realized.
2
12

-------
Appendix B
Page 5 of 7
DRAFT
Some states invested considerable resources in developing self-assessments about which
they received no EPA feedback.
Vision: What Should Performance Partnerships Be?
A Performance Partnership embraces a data-driven system of joint state/EPA planning
and priority setting that articulates: 1) federal and state government roles and
responsibilities; 2) environmental goals; 3) performance measures and time frames; and,
4) resource needs. Ultimately, it provides a framework that can guide delivery of
environmental services and environmental problem solving at all levels of government.
This data-driven joint state-EPA priority setting and planning process would yield a set of
agreed upon national, regional, and state and local priorities that build on the core
responsibilities of each. It would provide the information foundation for individual
states/Environmental Council Of the States (ECOS) to be further involved in improving
EPA's budgeting and accounting systems, strategic planning, and development of
environmental indicators and information management systems.
It would be an evolving, learning partnership between EPA and the states that rewards
innovation and innovators, encourages experimentation, and documents reduction in
transaction costs for both sides.
Joint EPA/state assessments of priorities and implementation strategies and annual or
biennial performance evaluations would take place to measure progress, as well as the
joint development of agreed upon performance measures.
Accountability, oversight and reporting would be based on a data-sharing relationship
between EPA and the states. From this we would jointly determine performance
measures, measures performance against them, provides feedback and continuously
improve them. Technological advancements that make information accessible to EPA,
states and the public would be readily adopted.
[For states with PPAs, jointly established priorities and plans for results would be
articulated in a single operating document ~ the performance partnership agreement.
This PPA would reflect the state's strategic plan. It would be a multi-media, integrated,
holistic agreement between the state and EPA that addresses environmental performance
expectations and needs. Criteria for what all such agreements should contain would be
developed by these states and EPA. In addition, EPA and states would identify incentives
3
13

-------
Appendix B
Page 6 of 7
DRAFT
and rewards for PPA states to continue to improve this process. Further Discussion
Necessary.]
A Performance Partnership Advancement Agenda: Overcoming the Challenges
To bridge some of the gaps between where performance partnerships are now and our vision for
the future, there are several areas that EPA and the states can begin to address.
Joint Priority Setting and Planning
The state/EPA performance partnership will be based on joint priority setting and
planning. Beginning the summer of 2002, in either a national forum or state-specific
regional processes, EPA and states will meet to identify a set of national, state and
regional priorities. These priorities will consider early information from our collective
environmental indicators and performance work. They will be incorporated into EPA's
national strategic planning process, and annual planning, budgeting and accountability
processes for FY 2004. States are encouraged to participate in these annual EPA
processes to ensure continuous improvement.
These priorities will continue to inform and refine the development of national and state
indicators, and the identification of investment needs for necessary data and monitoring
needs.
State priorities also will inform and influence the development of the FY 2004 Annual
Performance Goals, targets, and national program guidance, which in turn will encourage
flexible and innovative approaches to regional and state problem solving.
Joint Capacity Building Based on Performance Partnership Principles
EPA and the states jointly will review specific environmental program functions as
identified by EPA national program managers (i.e., NPDES, RCRA Permitting) in order
to improve efficiency and environmental impact of work. This would include a
renegotiation of roles and responsibilities for that specific function, as well as a review of
potential resource shifts.
Based on EPA's Innovation Action Agenda, EPA and states will actively search for
opportunities that display flexible and innovative solutions that produce measurable
environmental results. As regulatory or statutory barriers are identified and agreed upon,
they should be presented by the states and EPA to the EPA Administrator for further
action within an agreed upon time frame.
4
14

-------
Appendix B
Page 7 of 7
DRAFT
To improve the states' and EPA's ability to solve environmental problems, we should
provide incentives for continuous improvement and advancement of state and federal
programs to achieve environmental results. Among the incentives that could be used are:
o	Developing strategies for collection of necessary data and monitoring information
for development of environmental indicators. States and EPA jointly should
identify barriers to achieving results due to lack of data, including the need for
additional funding.
o	Accelerate development of system for shared data that would drive environmental
results.
o	Market the successes and benefits and establish acknowledgment and rewards for
long-term civil servants who support the cultural change.
o EPA and the states should provide opportunities and incentives for staff to engage in
continuous improvement. Among the incentives could be:
Joint state-EPA evaluation of employees/managers;
State and EPA senior leadership celebrating and sharing successes;
Successful state and federal employees mentoring others;
Regular state and EPA assessment of performance partnership
implementation;
More joint training opportunities for staff and senior managers;
Incorporating performance partnership principles into everyone's expected
job performance to include personal goal setting, performance evaluation
and incentives, both financial and other;
annual personnel performance reviews.
A joint annual evaluation of individual performance partnership agreements should be
institutionalized to ensure accountability and proper management. These evaluations
should include discussions of program performance, delegations and oversight
responsibilities as well as other kinds of appropriate process issues. EPA and state staff
should be accountable in performance evaluations for implementing performance
partnership agreements.
EPA and the states with PPAs may chose to perform joint performance assessments
annually or biennially in lieu of the self-assessments described in the May 17th
Performance Partnership Agreement.
5
15

-------
16

-------
Appendix C
Distribution
Headquarters Office
Associate Administrator for Congressional and Intergovernmental Relations (1301 A)
Deputy Associate Administrator for Intergovernmental Relations (1301 A)
Comptroller (2731 A)
Director, Grants Administration Division (3903R)
Director, Office of Regional Operations (1108 A)
Agency Followup Official (271 OA)
Agency Audit Followup Coordinator (2724A)
Audit Followup Coordinator, Office of the Administrator (1104)
Regional Office
Regional Administrators
Office of Inspector General
Inspector General
17

-------