A\^° S7>l^
& \
iSIKi
% PROI^
Offipp nf Tnspppfnr Cpnpral
Report of Audit
Grants
Improving Region 5fs EnPPA/PPG Program
Audit Report No. 2000-P-00008
February 29, 2000
-------
Inspector General Division
Conducting the Audit: Northern Audit Division
Chicago, Illinois
Region Covered:
Offices Involved:
Auditors:
Region 5
Office of Strategic Environmental Analysis
Air and Radiation Division
Resources Management Division
Superfund Division
Waste, Pesticides and Toxics Division
Water Division
Janet Norman
Robert Evans
-------
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
OFFICE OF THE INSPECTOR GENERAL
NORTHERN DIVISION
77 WEST JACKSON BOULEVARD
CHICAGO, IL 60604-3590
February 29, 2000
MEMORANDUM
SUBJECT: Final OIG Report No. 2000-P-00008
Improving Region 5's EnPPA/PPG Program
FROM: Charles M. Allberry
Acting Divisional Inspector General for Audits
Northern Division
TO:
Francis X. Lyons
Regional Administrator, Region 5
Attached is our final report on Region 5's Implementation and Oversight of the
Environmental Performance Partnership Agreements (EnPPA)/Performance Partnership Grants
(PPG) Program. The report contains recommendations and actions the Agency has taken or plans
to take. We appreciate your staffs' efforts in working with us during this evaluation and the
prompt action that you are taking to address the recommendations.
This audit report contains findings that describe problems the Office of Inspector General
(OIG) has identified and corrective actions the OIG recommends. This audit report represents the
opinion of the OIG and the findings contained in this audit report do not necessarily represent the
final EPA position. Final determinations on matters in this audit report will be made by EPA
managers in accordance with established EPA audit resolution procedures.
Action Required
In responding to the draft report, your office provided corrective actions, with milestone
dates, for each recommendation. Therefore, no further response is required, and we are closing
this report in our tracking system. Please track all corrective actions in the Management Audit
Tracking System.
We have no objections to the further release of this report to the public.
If you or your staff have any questions regarding the report, please contact me at (312)
353-4222 or Janet Norman, Senior Auditor, at (312) 886-0510.
-------
Improving Region 5's EnPPA/PPG Program
EXECUTIVE SUMMARY
INTRODUCTION
AND OBJECTIVES The U.S. Environmental Protection Agency (EPA) Region 5
awarded more than $45 million in fiscal year 1999 performance
partnership grants (PPG) to provide recipients flexibility, reduce
administrative burdens and costs, strengthen partnerships, and
improve environmental results. EPA's 1997 strategic plan states
that the President's "performance partnership" reinvention initiative
provides for increased flexibility in how a program is run in
exchange for increased accountability for results. A PPG is a multi-
program grant made to a recipient from funds allocated and
otherwise available for a specific environmental program. PPGs are
critical tools for implementing performance partnerships. An
Environmental Performance Partnership Agreement (EnPPA) is a
broad strategic document containing negotiated environmental
priorities and goals. The EnPPA may also include specific program
commitments that are incorporated by reference in the PPG.
Our audit objectives were to answer the following questions:
• Is Region 5 effectively implementing its EnPPA/PPG
program to ensure that recipients accomplish overall PPG
program goals?
Has Region 5 negotiated work plans that include adequate
accountability?
Does Region 5's oversight ensure that recipients meet work
plan commitments and use funds efficiently and effectively?
AREAS FOR
IMPROVEMENT AND Region 5's EnPPA/PPG program could benefit from improved
RECOMMENDATIONS communication. Increased use of environmental outcome
performance measures would help provide accountability.
Implementation of a regional post-award grant policy and an air
grant match calculation checklist will help provide financial
accountability. Finally, as part of our audit we looked at how
regional activities on EnPPAs/PPGs fit into the Government
i
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
Performance and Results Act (Results Act) goals and found that the
activities could come up under multiple goals.
Communication Key to
Success
Region 5 has not fully met stated goals of: (1) increased
flexibility, (2) reduced administrative burden, (3) strengthened
partnerships, and (4) improved environmental results. Causes
include: communication breakdowns between Region 5 and the
states and within the Region, and a lack of authority for some
Region 5 EnPPA team members to negotiate for their program.
One Region 5 state is evaluating continued participation due to
funding delays in a previous year. One tribe has chosen not to
continue to participate because the program did not meet their
needs. Finally, according to officials of another state, they have
chosen not to continue to participate due to funding delays and
state management changes.
Performance Measures
Linked to Accountability
Through the use of performance measures, Region 5 has negotiated
work plans that provide some accountability, but there is room for
improvement. EnPPAs are required to include core performance
measures. They also include regional measures and state measures.
All measures should meet four criteria, they should be: quantifiable,
verifiable, measurable, and time specific. The measures in the
agreements should also move toward focusing on outcome
measures and environmental results. We found that the quality of
measures in the work plans varied. Because outcome measures and
measures meeting the four criteria are the best indicators of
environmental results, it is difficult for regional officials to assess
whether progress is being made toward environmental results.
Improvements to Financial Regional oversight ensures that recipients accomplish work plan
Accountability objectives. However, limited post-award review of PPGs prevents
officials from being able to assess whether funds are used efficiently
and effectively. Post-award grants management was a fiscal year
1998 Agency level material weakness and Region 5 material
weakness. The Region 5 Acquisition and Assistance Branch
recently developed a written policy that, when implemented, should
provide additional financial accountability and increase the
likelihood that funds are used efficiently and effectively. Further
changes are needed to improve Region 5 program level confidence
in the work plans.
ii
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
Other Matters: EnPPAs/ EPA's Results Act structure allows for some activities supporting
PPGs and the Results Act EnPPAs/PPGs to be charged under more than one goal. The
Results Act was implemented to more accurately account for costs
associated with program goals and subobjectives. Activities to
develop EnPPAs/PPGs are multimedia, and there is no funding
allocated for EnPPA/PPG activities at the regional level. Also, due
to an inherent conflict between PPG reporting and the Results Act,
funding distributed to the states through PPGs will be reported
under the Results Act based on budgeted amounts, not actual
expenditures. Without changes, Agency reports of costs associated
with the PPG subobjective will be understated, and reports will be
misleading.
Details on each of these findings and recommendations are
contained in Chapters 2 through 5.
AGENCY COMMENTS
AND ACTIONS In response to the draft report, the Regional Administrator agreed
with the recommendations. Action plans with milestone dates were
also provided. A summary of the response and action plans is
included throughout the report, and a complete copy of the
response is included in appendix 1.
OIG EVALUATION
The Agency's proposed actions, when completed, will address the
findings and recommendations in this report.
in
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
[This page intentionally left blank.]
iv
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
Table of Contents
EXECUTIVE SUMMARY i
ABBREVIATIONS vii
CHAPTERS
1 INTRODUCTION 1
Purpose 1
Background 2
What is a PPG? 2
What is an EnPPA? 2
EnPPAs/PPGs and NEPPS 2
How many Region 5 states and tribes have a PPG and/or an EnPPA? ... 3
Region 5 EnPPA/PPG Program Organizational Structure 4
Scope and Methodology 5
Prior Audit Coverage 5
2 IMPLEMENTATION IMPROVEMENTS WILL HELP ACCOMPLISH PROGRAM
GOALS 7
Program Goals 7
Barriers to Meeting Goals 8
Results of Not Meeting Goals 10
Conclusion 10
Recommendations 11
Agency Comments and Actions 11
OIG Evaluation 12
3 BETTER MEASURES WILL HELP IMPROVE ACCOUNTABILITY 13
Guidance Defines Work Plan and Performance Measure Requirements 13
EnPPAs Provide Limited Accountability 14
Better Measures Can Improve Accountability 15
Conclusion 15
Recommendations 16
Agency Comments and Actions 16
OIG Evaluation 17
v
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
4 CHANGES NEEDED TO ASSESS USE OF FUNDS 18
Regulations and PPG Guidance Define EPA Grant Oversight 18
Region 5 Oversight 19
Difficult to Assess Program 20
Recommendations 20
Agency Comments and Actions 21
OIG Evaluation 21
5 OTHER MATTERS: CHANGES NEEDED TO MEET THE INTENT OF THE
RESULTS ACT 22
Results Act Requirements 22
EnPPA/PPG Program and Results Act Conflict 23
Recommendations 25
Agency Comments and Actions 25
OIG Evaluation 25
EXHIBIT
1 SCOPE AND METHODOLOGY 26
Effective Implementation 26
Adequate Accountability 26
Oversight 27
Other Steps 27
APPENDICES
1 REGION 5 REGIONAL ADMINISTRATOR'S RESPONSE TO THE DRAFT
REPORT 28
2 DISTRIBUTION 35
vi
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
ABBREVIATIONS
EnPPA
Environmental Performance Partnership Agreement
EPA
U.S. Environmental Protection Agency
IEPA
Illinois Environmental Protection Agency
MPCA
Minnesota Pollution Control Agency
NEPPS
National Environmental Performance Partnership System
OEPA
Ohio Environmental Protection Agency
OIG
Office of Inspector General
PPG
Performance Partnership Grant
QAPP
Quality Assurance Program Plan
Results Act
Government Performance and Results Act (also known as GPRA)
vii
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
[This page intentionally left blank.]
viii
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
CHAPTER 1
Introduction
PURPOSE
The U.S. Environmental Protection Agency's (EPA) 1997 strategic
plan states that the President's "performance partnership"
reinvention initiative provides for increased flexibility in how a
program is run in exchange for increased accountability for results.
Performance partnerships are part of EPA's approach to working
with states and tribes (recipients) to better implement
environmental programs and achieve environmental results. A
performance partnership grant (PPG) is a critical tool for EPA's
implementation of performance partnerships. Both performance
partnerships and PPGs represent a significant shift in how EPA
and its partners work together to address human health and
environmental protection.
This audit is part of the Office of Inspector General's (OIG) PPGs
and National Environmental Performance Partnership System
Issue Area Plan, dated September 1998. The OIG made a
commitment to the Agency to assess EPA's PPG program
implementation and determine what improvements were needed to
ensure EPA implemented the program effectively and achieved
environmental results.
Our audit objectives were to answer the following questions:
• Is Region 5 effectively implementing its EnPPA/PPG
program to ensure that recipients accomplish overall PPG
program goals?
Has Region 5 negotiated work plans that include adequate
accountability?
Does Region 5's oversight ensure that recipients meet
work plan commitments and use funds efficiently and
effectively?
1
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
A PPG is a multi-program grant made to a recipient from funds
allocated and otherwise available for a specific environmental
program. EPA developed PPGs in response to recommendations
from internal and external stakeholders to: (1) increase recipient
flexibility, (2) achieve administrative savings by streamlining the
grants process, (3) strengthen partnerships, and (4) help recipients
improve environmental results. These recommendations formed
the basis for the PPG program's purposes and goals. Beginning in
fiscal year 1996, EPA received authority to award PPGs.
What is an EnPPA? An environmental performance partnership agreement (EnPPA) is
an agreement between a region and a state that describes jointly
developed goals, objectives, and priorities. The agreements
typically include the strategies for meeting commitments, the roles
and responsibilities of the state and EPA, and the measures for
assessing progress. PPG program commitments must be outlined
in a work plan, and most Region 5 states that opt to use a PPG
often have the EnPPA serve as the PPG work plan.
EnPPAs/PPGs On May 17, 1995, state and EPA leaders signed a Joint
and NEPPS Commitment to Reform Oversight and Create a National
Environmental Performance Partnership System. The objective of
this agreement was to accelerate the transition to a new working
relationship between EPA and states-one which reflected the
advancement made in environmental protection over the preceding
two decades by both states and EPA. In addition, the agreement
recognized that existing policies and management approaches must
be modified to ensure continued environmental progress.
EPA PPG guidance, dated October 6, 1998, states that the key
goals this new partnership agreement shares with PPGs are:
to allow States and EPA to achieve improved
environmental results by directing scarce public
resources toward the highest priority, highest value
activities; to provide States with greater flexibility
to achieve those results; to improve public
understanding of environmental conditions and
2
BACKGROUND
What is a PPG?
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
choices; and to enhance accountability to the public
and taxpayers.
While the National Environmental Performance Partnership
System (NEPPS) was an agreement between states and EPA, many
concepts of this performance partnership also apply to EPA's
relationship with tribes. Under performance partnerships, EPA
and its partners are expected to achieve more integrated
environmental management and enhanced environmental results.
This new partnership should also enable EPA and its partners to
move beyond relying solely on numbers of permits issued,
inspections made, or similar measures, to performance measures
that more directly reflect changes in environmental quality.
How many Region 5 states
and tribes have a PPG
and/or an EnPPA?
In fiscal year 2000, four of the six states in Region 5 have PPGs
and EnPPAs1. In fiscal year 1999, Region 5 awarded more than
$45 million in PPGs to four state environmental programs and
three state pesticide programs. Another Region 5 state had an
EnPPA, but no PPG. The state environmental programs all used
an EnPPA to serve as the PPG work plan. The state pesticide
programs do not use EnPPAs, and we did not review these PPGs.
In fiscal year 1998, one Region 5 tribe had a PPG only (tribes do
not use EnPPAs).
Region 5 consists of six states-Illinois, Indiana, Michigan, Minnesota, Ohio, and Wisconsin. Twenty-
five of the 35 Federally recognized tribes in Region 5 have two or more continuing grant programs that could be
combined into a PPG.
3
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
Table 1:
Region 5 States and Tribes wit
l PPGs and/or EnPPAs
2000
1999
1998
Illinois
Both
Both
Both
Indiana
Both
Both
Both
Minnesota
Both
Both
Both
Ohio
None
Both
Both
Wisconsin
Both
EnPPA only
EnPPA only
Fond du Lac
None
None
PPG only
Region 5 EnPPA/PPG EPA developed interim guidance in 1996 and 1998 that served as
Program Organizational the operating guidance for states and tribes interested in applying
Structure for PPGs. Regional offices were to use this operating guidance to
develop and implement their EnPPA/PPG programs.
Region 5 established a NEPPS coordinator in the Office of
Strategic Environmental Analysis. The NEPPS coordinator leads
the NEPPS workgroup, which consists of one team leader
representing each Region 5 state with an EnPPA and/or PPG,
representatives of the Resources Management Division (grants),
the Office of Regional Counsel, and the Office of Inspector
General in an advisory role. The Region 5 state EnPPA team
leaders are responsible for overseeing and coordinating the
development of the EnPPAs for their respective states. The team
leaders have teams that consist of one representative from each
program office. These team members are usually project officers
who then coordinate negotiation issues with program technical
staff. The system was designed to leave major programmatic
negotiations and decisions in the programs, where the expertise on
environmental issues and program management remains.
Regional grants specialists are responsible for PPG administrative
functions. Region 5's grants specialists are responsible for
4
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
reviewing PPG applications and related forms and certifications,
preparing documentation related to the grant award and
amendments, reviewing financial status reports, and obtaining
verification of recipients' PPG funding match.
SCOPE AND
METHODOLOGY We performed our audit in accordance with generally accepted
government auditing standards. We conducted our fieldwork from
July 1999 through November 1999. We specifically reviewed three
states' and one tribe's EnPPA and/or PPG (for time periods
beginning in fiscal year 1998), and the associated self-assessments,
and annual performance reports.
We performed our fieldwork at Region 5 in Chicago, Illinois. We
also visited the Illinois Environmental Protection Agency in
Springfield, Illinois; and the Fond du Lac Reservation in
Minnesota.
For further details on scope and methodology, see exhibit 1.
The OIG has not issued any audit reports related to Region 5's
EnPPA/PPG program implementation. However, the OIG
conducted a joint management assistance review with Region 5
staff of Indiana's 1997 PPG and EnPPA and the negotiation
process for the 1998 PPG and EnPPA. As a result of the
management assistance review, OIG and regional staff suggested
areas for improvement that included:
communication-overall, to reach agreement upon level of
detail in the document, to distribute grant guidance, and on
enforcement;
continuity of the process among state managers;
state's ability to meet reporting commitments;
development and inclusion of performance
measures/indicators; and
state financial system changes to promote flexibility.
PRIOR AUDIT
COVERAGE
5
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
The OIG has also served on the regional EnPPA teams in an
advisory role. Our goals were to ensure the region implemented
proper policies and procedures for NEPPS and to identify any
barriers or problems. In this capacity, we have reviewed the
EnPPAs prior to regional approval to (1) assess whether the work
plans were consistent with the intent of NEPPS, and (2) determine
whether the documents serving as PPG work plans met the basic
requirements of PPG guidance.
The U.S. General Accounting Office issued a report,
ENVIRONMENTAL PROTECTION: Collaborative EPA-State
Effort Needed to Improve New Performance Partnership System,
dated June 21, 1999. GAO findings included:
The development of outcome measures was impeded by an
absence of baseline data, inherent difficulty in quantifying
certain results, difficulty of linking program activities to
environmental results, and considerable resources needed
for high-quality performance measurement.
Factors that complicated EPA's oversight included: (1)
statutory and/or regulatory requirements in some cases
prescribed the kind of oversight required of states by EPA,
(2) reluctance by EPA staff to reduce oversight without the
measures in place to ensure that environmental quality
would not be compromised, and (3) the inherent difficulty in
"letting go" on the part of some EPA staff that have
implemented the existing EPA-state oversight arrangement
for years.
There were a number of reported benefits associated with
the new system that included (1) improving communication
about program priorities among EPA and state program
staff and (2) allowing states the option to shift resources
under the PPG program.
6
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
CHAPTER 2
Implementation Improvements
^^Olel^Accomglisl^rograii^oal^
Region 5 has not fully met stated goals of: (1) increased
flexibility, (2) reduced administrative burden, (3) strengthened
partnerships, and (4) improved environmental results. Causes
include: communication breakdowns between Region 5 and the
states and within the Region, and a lack of authority for some
Region 5 EnPPA team members to negotiate for their program.
One Region 5 state is evaluating continued participation due to
funding delays in a previous year. One tribe has chosen not to
continue to participate because the program did not meet their
needs. Finally, according to officials of another state, they have
chosen not to continue to participate due to funding delays and
state management changes.
PROGRAM GOALS
EPA guidance, dated July 1996 and October 6, 1998, describes
four goals of the PPG program.
Flexibility-Recipients will have the flexibility to address
their highest environmental priorities, while continuing to
address core program requirements.
Administrative Savings-Recipients and EPA can reduce
administrative burdens and costs by reducing the number of
grant applications, budgets, work plans, and reports.
Strengthened Partnerships-EPA will develop
partnerships with recipients where both parties share the
same environmental and program goals and deploy their
unique resources and abilities to jointly accomplish those
goals.
Improved Environmental Results-Recipients can
improve environmental results and more effectively link
7
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
program activities with environmental goals and program
outcomes; and develop innovative pollution prevention,
ecosystem, and community-based strategies.
BARRIERS TO
MEETING GOALS Communication breakdowns adversely affected Region 5's
partnership with two states and one tribe. For example, the Ohio
Environmental Protection Agency (OEPA) and Region 5 did not
communicate adequately to resolve a water program quality
assurance program plan (QAPP) issue, causing funding delays.
Illinois Environmental Protection Agency (IEPA) also experienced
funding delays that were a result of poor communication on an
over allocation of water funds. Lastly, the Fond du Lac tribe
experienced communication problems with a QAPP that nearly
caused funding delays.
• Region 5 officials did not inform OEPA officials that there
were problems with the OEPA Water Program QAPP.
Region 5 officials elected to withhold all PPG funds,
pending resolution of the QAPP issue, even though this
portion of the grant represented only 21 percent of the total
PPG. OEPA officials eventually received the PPG funds in
the last month of the grant period when the state was
coming close to not meeting payroll. Region 5 water
program officials took steps to address QAPP approval
delays and communication problems. Regional officials
included a term and condition as part of the fiscal year
1999 PPG to address quality assurance issues with the
Clean Water Act Section 106 program and initiated a
process to improve communications among programs and
states.
Region 5 officials did not communicate the seriousness of
an over allocation of water program funds in IEPA's fiscal
year 1999 application. Due to a discrepancy of $16,500
between the application amount for the Public Water
Supply System program and the actual allotment from U.S.
EPA Headquarters, regional officials could not release the
total PPG funds of about $16 million. When IEPA learned
8
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
about the discrepancy, and that it would prevent the release
of funds, IEPA was already in jeopardy of not meeting
payroll. At the beginning of the fiscal year 2000 EnPPA
development process, the Regional Administrator
committed to IEPA that communication will be better.
Both IEPA and regional officials agreed to work to keep
each other better informed of problems and to elevate
issues earlier.
Fond du Lac tribal officials submitted a QAPP to Region 5
and it was not approved for 18 months. Region 5
threatened to deny a grant extension and withhold all of the
tribe's PPG funds pending approval of the QAPP, even
though the QAPP affected only a portion of the total grant
funds. The QAPP was not approved until a Region 5
Assistance Agreement Branch official intervened resulting
in the release of funds to the tribe.
Some barriers to reaching program goals are beyond Region 5's
control. For example, state expectations have often exceeded what
Region 5 has been able to meet. Many state officials expected
considerable reductions in oversight and reporting activities.
Regional officials had already taken steps to reduce reporting
requirements prior to the introduction of NEPPS. Region 5
officials were therefore limited in what remained that could be
reduced, due to regulations, reporting requirements, and statutory
requirements. Some states are also limited in flexibility due to
state financial systems or the political climate.
Negotiation of enforcement language has continued to be difficult
for Region 5 and the states. For example, Wisconsin and Region 5
officials were in negotiations for months on general enforcement
language that outlined the state and Federal enforcement roles, and
specific enforcement commitments for the air and RCRA
programs. State officials did not agree with the characterization of
the Federal enforcement role. Later, state and regional officials
did not agree on the level of detail and commitments for the air
and RCRA programs.
9
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
RESULTS OF NOT
MEETING GOALS One state and one tribe have dropped out of the PPG program.
Another state is evaluating its continued participation. These
states and the tribe have realized limited improvements in
partnership with Region 5. Michigan, the only Region 5 state that
has not tried an EnPPA or a PPG, is not participating until state
officials see that the benefits of the EnPPA/PPG program
outweigh the costs. As a result of the problems identified above,
Ohio has withdrawn from the program due to funding delays and
OEPA management changes. Illinois has threatened to withdraw
from the program, if Region 5 does not make improvements to
ensure funds are received timely. Wisconsin threatened to
withdraw from the program during enforcement language and
commitment negotiations, but now has a finalized agreement. The
Minnesota Fond du Lac tribe has withdrawn from the program
because their expectations were not met. Tribal officials saw little
improvement in partnership with Region 5. Also, though the tribe
was able to experience some flexibility benefits, it was not to the
extent expected, based on experience with tribal grants from other
Federal entities.
The IEPA has been able to make changes to help realize the
benefits of flexibility. The previous IEPA Director instructed each
program to set aside 5.5 percent of Federal funds for pollution
prevention and regulatory innovation programs. The pollution
prevention program funded an intern program where interns
worked on-site with companies to identify pollution prevention
opportunities. Due to the success of the program, many companies
were eager to participate.
CONCLUSION
Not all of the benefits that EPA and states expected from the
EnPPA/PPG program were realized, especially the goal of
improved environmental results. Some of this was the
responsibility of Region 5 officials, but others were based on high
state expectations. Some states also haven't taken advantage of the
opportunity to shift funds, to the extent expected. Working
together in the future, state and regional officials can learn from
10
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
past experiences and continue efforts to work together and
improve partnerships.
RECOMMENDATIONS
We recommend the Regional Administrator:
2-1. Develop and document an action plan, with milestone dates
and detailed descriptions of the Region 5 water program
plan to improve the QAPP process (approval and
communication).
2-2. Develop and document a process to elevate EnPPA/PPG
concerns to the Region 5 Senior Leadership Team. For
example, if one office or division leading negotiations
makes a decision that could adversely impact the entire
EnPPA/PPG (such as withholding funding), that issue
should be brought to the attention of senior regional
management.
2-3. Require regional officials to maintain open communication
with state officials after an EnPPA is signed, and until
grant payments are issued. For example, regional EnPPA
team leaders should work more closely with regional grant
program officials to ensure the state receives funds timely
and identify early anything that may affect timely receipt
of funds.
2-4. Work with participating states to develop a strategy to
overcome barriers to accomplishing the four PPG program
goals. Promote information sharing of best practices
among states, for example, obtain examples from other
regions and states on their successful efforts.
AGENCY COMMENTS
AND ACTIONS The Regional Administrator concurred with the recommendations.
The Water Division, along with the Resources Management
Division, have taken steps to improve the quality assurance
process. Water Division officials will work with all interested
11
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
parties to conduct an in-depth review of the events that occurred
surrounding the Fond du Lac tribe QAPP.
The document entitled "Region 5 EnPPA/PPG Process,"
establishes roles and responsibilities, including when issues need
to be brought to the attention of upper management. The Region
will continue to stress the need for coordination and elevation of
issues as needed. The Region will also amend the document by
March 31, 2000 to include a discussion of the need to continue
discussions with a state during the period between finalizing the
EnPPA and payment of the grant. The State/EPA Grants
Workgroup has also discussed this issue and is developing a
regional policy to ensure states receive funds as soon as possible.
The Region will hold another meeting with participating NEPPS
states during 2000 to discuss best practices and development of
environmental outcome measures.
OIG EVALUATION
The proposed actions, when implemented will address the findings
and recommendations.
12
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
CHAPTER 3
Better Measures Will Help
Through the use of performance measures, Region 5 has
negotiated work plans that provide some accountability, but there
is room for improvement. EnPPAs are required to include core
performance measures. They also include regional measures and
state measures. All measures should meet four criteria, they
should be: quantifiable, verifiable, measurable, and time specific.
The measures in the agreements should also move toward focusing
on outcome measures and environmental results. We found that
the quality of measures in the work plans varied. Because
outcome measures and measures meeting the four criteria are the
best indicators of environmental results, it is difficult for regional
officials to assess whether progress is being made toward
environmental results.
GUIDANCE DEFINES
WORK PLAN AND
PERFORMANCE
MEASURE
REQUIREMENTS
The 1996 PPG guidance outlines PPG work plan requirements as
the basis for the management and evaluation of performance under
the grant agreement. The PPG work plan should include core
program commitments (goals, performance measures, program
activities) derived from statutes, regulations, and standing legal
agreements between EPA and recipients. Performance measures
that are PPG program commitments must be time-specific,
measurable, quantifiable, and verifiable. In addition, EPA
encourages all recipients to adopt outcome and output-oriented
performance measures that track program performance and
environmental conditions and trends.
Core performance measures are a limited set of national measures
designed to help gauge progress toward protection of the
environment and human health and provide a national picture of
the status of the environment. EPA and the Environmental
Council of the States developed core performance measures to
13
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
help focus EPA and state attention on improving how they measure
the effectiveness of environmental protection efforts.
Appropriate accountability provisions are essential in designing the
new PPG program. The PPG guidance states that a fundamental
goal of EPA's efforts to design accountability provisions into
PPGs is to begin moving Federal, state, and tribal programs toward
the use of results-oriented measures of environmental and program
performance that are understandable and meaningful to the public.
Further, the work plan should also establish procedures (e.g.,
midyear and end-of-year reviews, reporting requirements, joint
activities) that EPA and the recipient will use for evaluating
accomplishments, discussing progress, and making adjustments to
meet milestones. The PPG guidance states that EPA should work
with recipients to balance the need to maintain core program
requirements with the need to incorporate program flexibility and
move toward program performance measures and environmental
indicators.
EnPPAs PROVIDE
LIMITED ACCOUNT- The fiscal year 1998 EnPPAs for Illinois, Indiana, and Minnesota
ABILITY we reviewed included core performance measures, specifically or
by reference. About 50 percent of performance measures in the
agreements met all four criteria (quantifiable, verifiable,
measurable, and time specific). The remaining performance
measures, which included measures for state funded activities, did
not meet at least one of the four criteria. We also noted that
Illinois and Minnesota EnPPAs clearly showed a move toward
outcome measures. Wisconsin's EnPPA was largely based on
activity measures, with virtually no outcome measures.
Program and grant officials reviewed costs for reasonableness with
commitments prior to award and approval of grant applications.
Our review showed that PPG budgets were tied to EnPPA
commitments.
14
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
BETTER MEASURES
CAN IMPROVE Development of performance measures is a difficult and resource
ACCOUNTABILITY intensive task. State capacity to develop performance measures
varies. For example, Minnesota had 537 measures, 63 percent of
which met all four criteria (quantifiable, verifiable, measurable,
and time specific), Wisconsin had 180 measures, 29 percent of
which met all four criteria. Many of Minnesota's measures are to
satisfy certain state reporting requirements. The quantity alone is
not an indication of the quality of the measures. However, the
quantity of measures does indicate the time and effort a state has
devoted to developing measures.
Region 5 needs an objective basis to evaluate state environmental
programs. Without better measures, regional officials have a
difficult time assessing environmental results due to variations
among states. Even with the minimum core performance measures
there will still be varying abilities and information among states.
Regional officials have no consistent and comprehensive basis to
determine whether environmental programs are moving toward
achieving environmental results. Regional officials need to be able
to make an assessment of state programs, identify weaknesses, and
hold states accountable for improving those weaknesses. Without
ramifications, such as withholding funding, there is no incentive
for states with weak programs to improve.
CONCLUSION
Region 5 EnPPAs provide some accountability but there is room
for improvement. Some Region 5 EnPPAs we reviewed included
commitments that were primarily output measures. Although
activity or output measures can provide valuable information, they
do not reflect environmental improvements and results. Outcome
measures with appropriate indicators and baseline data are needed
to adequately assess environmental results. The EPA's view is
that a mix of outputs and environmental measures is necessary to
demonstrate accountability under NEPPS. Region 5 should
continue efforts to work with recipients in developing good
outcome measures that are time specific, quantifiable, measurable,
15
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
and verifiable. Without EnPPAs that clearly demonstrate adequate
accountability, the Region cannot use them to hold recipients
accountable.
RECOMMENDATIONS
We recommend that the Regional Administrator:
3-1. Require regional EnPPA team members to review
performance measures during negotiations to evaluate
whether measures are time specific, measurable,
quantifiable, and verifiable. Team members should work
with state officials to develop measures that meet these
four criteria.
3-2. Instruct the Senior Leadership Team to encourage National
Program Managers and regional program staff to move
more toward outcome measures rather than just activity or
output measures.
3-3. Require regional program managers to make development
of outcome measures with recipients a priority. For
example, provide EnPPA team members with the resources
to develop and seek out good examples of outcome
measures for reference by all regional and recipient staff.
AGENCY COMMENTS
AND ACTIONS The Region concurred with these recommendations and will amend
the "Region 5 EnPPA/PPG Process" document by March 31, 2000
to emphasize the need to evaluate whether proposed measures meet
the four criteria. The Region acknowledges that it has little control
over the required Core Performance Measures.
The Region agrees that development of outcome measures is a
priority and will address this topic in the next meeting with all
Region 5 NEPPS states. The Region 5 NEPPS workgroup will
also undertake a project to find good examples of outcome
measures to present at this meeting.
16
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
OIG EVALUATION
The Region's proposed actions, when implemented will address the
findings and recommendations.
17
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
CHAPTER 4
Changes Needed to Assess Use of Funds
Regional oversight ensures that recipients accomplish work plan
objectives. However, limited post-award review of PPGs prevents
officials from being able to assess whether funds are used
efficiently and effectively. Post-award grants management was a
fiscal year 1998 Agency level material weakness and Region 5
material weakness. The Region 5 Acquisition and Assistance
Branch recently developed a written policy that, when
implemented, should provide additional financial accountability
and increase the likelihood that funds are used efficiently and
effectively. Further changes are needed to improve Region 5
program level confidence in the work plans.
REGULATIONS AND
PPG GUIDANCE Regulations require PPG recipients to prepare performance
reports.
DEFINE EPA GRANT PPG project officers are required to review the reports and provide
OVERSIGHT the recipient with an evaluation of findings. As part of the
Region's and recipients' negotiation of work plan commitments,
PPG guidance states that the Region and recipients should
establish procedures that EPA and the recipient will use for
evaluating accomplishments, discussing progress, and making
adjustments to meet milestones.
PPG guidance established procedures for financial management
and reporting. For example, recipient fiscal control and
accounting procedures must be sufficient to permit preparation of
Financial Status Reports for PPG awards. PPG recipients must
maintain accounting and financial records that adequately identify
the source (i.e., federal funds and match) and application of funds
provided for PPG activities. These records should contain relevant
information such as obligations, unobligated balances, outlays,
expenditures, and program income. However, PPG costs will not
be tracked to each of the original individual categorical source(s)
of grant funding.
18
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
PPG guidance also includes detailed information on match
calculations. The required cost share (based on the match or
maintenance of effort requirements of the categorical grants
included in the PPG) will be the same under PPGs as under
categorical grants. However, EPA may determine that there are
exceptional circumstances justifying a reduction in cost share for a
PPG for the year that the PPG is awarded. The recipient's
minimum cost share requirement applies to the entire grant.
Recipients must maintain adequate financial records of their cost
share.
REGION 5 OVERSIGHT
Region 5 officials did a good job of using year-end performance
reports and self-assessment reports. Programs used state midyear,
year-end, and self assessment reports as an oversight tool to
evaluate whether states and EPA accomplished goals. Region 5
officials also did a good job of oversight as part of day-to-day
interactions with states. This has changed little since the
introduction of EnPPAs/PPGs.
Grant officials have focused resources on tribal grant issues. To
help ensure that state grant issues are not ignored, Acquisition and
Assistance Branch officials recently developed a post-award grant
policy that, when implemented, will: (1) provide more financial
accountability, (2) assure that grant funds are spent appropriately,
and (3) address the post-award process material weaknesses. For
example, the new policy calls for a regional team, consisting of
resource management division and program officials, to visit the
recipients every two years and provide guidance and assistance as
needed2.
The state match calculations for Illinois, Indiana, Minnesota, and
Ohio were accurate for all programs, except the air program. We
2According to the policy, recipients will be categorized and selected by award amount and type. The three
categories are: under $100,000, $101,000 to $1 million, and over $1 million. Recipients will also be categorized
by state, tribe and other. At least two recipients in each category will be selected for these administrative/financial
reviews. Sites selected for visits will be based on recipient performance and compliance with grant requirements.
19
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
were unable to determine if the air program state match amounts
were accurate because the air program project officers did not use
the same calculation method when calculating state match
amounts. However, the states all matched at least 40 percent and
there was no financial impact. Regional officials have recently
developed a calculation sheet that will help ensure consistency in
match calculations for future PPGs and clarify the match
calculation requirements. Air officials also believe the proposed
revisions to 40 CFR Part 35 will provide clearer guidance on how
to calculate the state match.
Some Region 5 officials have a lack of confidence in the
accountability provided by the EnPPA/PPG process. For example,
some regional officials see the reduced reporting of commitments
in the EnPPAs as a drastic reduction in accountability. This made
it difficult for these officials to support the EnPPA/PPG
negotiation process.
DIFFICULT TO ASSESS
PROGRAM It is difficult for regional officials to assess whether states are
spending PPG funds efficiently and effectively. Changes are
needed to improve regional staff member confidence in the
accountability provided by EnPPAs/PPGs. Future implementation
of the post-award grant policy will help provide financial
accountability. Finally, air program staff use of the calculation
sheet will help ensure that the match is calculated consistently for
all Region 5 states.
RECOMMENDATIONS
We recommend the Regional Administrator:
4-1. Develop regional guidance on what, at a minimum, should
be included in an EnPPA to provide accountability.
4-2. Hold staff accountable for implementing the Acquisition
and Assistance Branch post-award grant policy.
20
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
4-3. Require air program staff to use the calculation sheet until
revisions to 40 CFR Part 35 are finalized.
AGENCY COMMENTS
AND ACTIONS The Region concurred with the recommendations. The Region 5
NEPPS Workgroup will hold informational sessions with each
program office before August 31, 2000 to encourage a common
approach to determining whether measures (and EnPPAs) provide a
minimum standard of accountability.
OIG EVALUATION
The Region's actions, when implemented will address our findings
and recommendations.
21
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
CHAPTER 5
Other Matters: Changes Needed
to Meet the Intent of the Results Act
EPA's Results Act structure allows for some activities supporting
EnPPAs/PPGs to be charged under more than one goal. The
Results Act was implemented to more accurately account for costs
associated with program goals and subobjectives. Activities to
develop EnPPAs/PPGs are multimedia, and there is no funding
allocated for EnPPA/PPG activities at the regional level. Also, due
to an inherent conflict between PPG reporting and the Results Act,
funding distributed to the states through PPGs will be reported
under the Results Act based on budgeted amounts, not actual
expenditures. Without changes, Agency reports of costs associated
with the PPG subobjective will be understated, and reports will be
misleading.
RESULTS ACT
REQUIREMENTS The Results Act established a system to set goals for program
performance and to measure results. The intent is to improve the
efficiency and effectiveness of Government programs. EPA
restructured the budget in 1999 to link activities and resources
using a new system of goals, objectives, and subobjectives. EPA
used the same structure to implement Government cost accounting
standards to help ensure the integration of planning, budgeting, and
accountability. Under this system, subobjectives are used to
account for costs.
The EnPPA/PPG program has bounced around between several
Results Act objectives and subobjectives. Most recently, the
program is included under:
Goal 7-Expansion of Americans' Right to Know About
their Environment,
Objective 3-Enhance Ability to Protect Public
Health, and
22
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
Subobjective 4-Local and Small
Government Outreach.
This is a Headquarters only objective and subobjective. There is
no funding associated with the objective and subobjective at the
regional level.
The Senate recently reported concerns that funding for key
programs often is divided into several goals or objectives with
little rationale for how the funding is allocated. Many program
activities easily could be justified under several goals or
objectives, leading to serious questions about budget
accountability. For example, regional official time spent on
development of EnPPAs/PPGs could fit under goals and
subobjectives for the various media programs, or the PPG
development subobjective.
The Results Act requires the Agency to issue its first performance
report using the new budget structure in March 2000. In reporting
on the cost of the activities related to the PPG subobjective, the
Agency needs to describe accurately the activities associated with
the subobjective. To prepare accurate reports under the Results
Act, EPA needs to ensure the budget structure accurately reflects
the activities and costs associated with development of
EnPPAs/PPGs.
When Congress first established the Results Act, one goal was to
increase the ability to determine program costs. For example, all
activities and costs charged to EPA's clean air goal should give an
indication how much it costs EPA to carry out an air program.
Congressional officials expect Results Act reports to indicate how
much a given program or goal costs.
EnPPA/PPG PROGRAM
AND RESULTS ACT Regional officials working on EnPPAs/PPGs are not associated
CONFLICT with the Results Act goal that reflects the work they actually do.
One official, who spent all of her time on EnPPAs/PPGs, found
that her time was charged to enforcement. Another official, whose
time was charged to Superfund, led development of the Wisconsin
23
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
agreement. Other EnPPA team leaders spending time on
EnPPA/PPG development, which is essentially multimedia, also
represented only their media program from an accounting
perspective. An OIG audit in Region 8, found that regional
officials allocated time spent on EnPPAs/PPGs evenly across all
media programs using a cost pool.
An inherent conflict exists between the reduced reporting
requirements of the EnPPA/PPG program and the additional data
required to support the Results Act. The Results Act requires
reporting on an activity level that PPG guidance does not require.
Guidance does not require states to report how the money was
spent for each program. This has been a source of great concern to
both state and regional officials. State officials worry that regional
officials will demand the detailed information. Regional officials
worry that the Results Act reporting will be misleading. To
address these concerns, the Region 5 Resource Conservation and
Recovery Act program has created a cross-walk between Results
Act activities and core performance measures. Regional officials
are also allocating PPG fund draw-downs across programs in the
same proportion as funds were originally awarded. For example,
if the air, water and waste programs represented 35 percent, 35
percent, and 30 percent (respectively) of the original grant award,
the PPG fund draw-downs would be allocated along the same
percentages.
Without changes, EPA reports under the Results Act will not
correctly match accomplishments to the level of effort (i.e.,
resources) used. EPA will understate the costs for the PPG
subobjective, because the costs will not reflect regional official
time spent on EnPPA/PPG development. Reporting under the
Results Act will also be misleading because some information will
be based on budgeted costs only, not actual expenditures as
implied.
24
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
RECOMMENDATIONS
We recommend the Regional Administrator:
5-1. Work with Office of the Chief Financial Officer officials to
determine the best placement of regional work on
EnPPAs/PPGs within the Results Act.
5-2. Consider ways to allocate time spent on development of
EnPPAs/PPGs within the region, so that reports accurately
depict activities and the associated goals.
5-3. Clearly indicate in reports that feed Results Act reporting
that the information presented is based on budgeted
amounts, not actual expenditures.
The Region concurred with these recommendations. Specifically,
the Region will continue to work with the Office of the Chief
Financial Officer and participate on the national planning network
to assess and improve the Agency's budget and accounting system
Regional officials believe that regional work on EnPPAs/PPGs
should remain within regional programs, placing resources within
the environmental goals they support as much as possible.
The Region will also implement a periodic assessment of the
amount of time people spend working on aspects of the NEPPS
program (for example: negotiation, processing, and follow-up).
The Region acknowledges that it may need to conform to
nationally-made decisions on portraying Results Act information.
OIG EVALUATION
The Region's actions, when implemented will address the findings
and recommendations.
AGENCY COMMENTS
AND ACTIONS
25
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
Exhibit 1
Page 1 of 2
EFFECTIVE
IMPLEMENTATION Our first objective was to determine if Region 5 effectively
implemented the EnPPA/PPG program to ensure that recipients
accomplished overall PPG program goals. Our review of Region
5's EnPPA/PPG program included:
(1) Interviewing Region 5's NEPPS Coordinator, EnPPA Team
Leaders, and PPG project officers regarding their roles and
responsibilities related to the EnPPA/PPG program.
(2) Reviewing national and regional guidance and policy to
determine program requirements.
(3) Interviewing officials representing all Region 5 states and one
tribe (Minnesota Fond du Lac Band of Chippewa).
ADEQUATE
ACCOUNTABILITY Our second objective was to determine if Region 5 negotiated
work plans that included adequate accountability. Our review of
recipients' PPG work plans included:
(1) Reviewing criteria describing the elements of a work plan.
(2) Interviewing regional officials.
(3) Selecting and reviewing a judgmental sample of EnPPAs. We
selected the 1998 EnPPAs from Illinois, Minnesota, and
Wisconsin. We did not select Ohio because state officials had
recently decided not to continue. We did not select Indiana
because the state had previously been reviewed as part of the joint
OIG/Region 5 management assessment review.
26
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
Exhibit 1
Page 2 of 2
OVERSIGHT
Our third objective was to determine if Region 5's oversight
ensured that recipients met work plan commitments and used
funds efficiently and effectively. Our review of Region 5's
oversight included:
(1) Reviewing criteria that described the appropriate use of funds,
specifically for PPGs.
(2) Interviewing regional officials.
(3) Reviewing a sample of PPG files.
(4) Selecting and reviewing a sample of EnPPAs and self-
assessments. We reviewed the same EnPPAs identified for
objective 2 and the corresponding self-assessments.
OTHER STEPS
In addition to our audit objectives, we conducted steps to assess
the EnPPA/PPG program placement within GPRA. We also
looked at internal controls by reviewing Federal Managers'
Financial Integrity Act reports. Finally, we interviewed regional
officials and reviewed documents to learn how the EnPPA/PPG
program fit into regional operations.
27
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
Appendix 1
Page 1 of 7
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
REGION 5
77 WEST JACKSON BOULEVARD
CHICAGO, IL 60604-3590
FEB 14 2000
REPLY TO THE ATTENTION OF:
R-19J
MEMORANDUM
SUBJECT: Draft OIG Report No. OOXXXXXX
Improving Region 5's EnPPA/PPG Program
FROM:
Francis X. Lyons
Regional Administrator
TO:
Charles M. Allberry
Acting Divisional Inspector General for Audits
Northern Region
Thank you for the opportunity to comment on the draft report on Region 5's Oversight of the
Environmental Performance Partnership Agreements (EnPPA) Performance Partnership Grants
(PPG) Program. Since the Region undertook performance partnerships in 1995 we have
endeavored to continually improve our approach to this new system. In short, we want to get
environmental results, we want to measure the results, and we want to improve the process by
which we work with states and release funds to get these results. I hope you agree that our
efforts have been worthwhile.
Still, as the report notes, barriers remain within the Region and among our headquarters and
state agency partners. We hope the attached comments on the draft report will help hone the
review your staff has conducted and clarify certain issues raised in the draft report. The response
includes comments on the factual accuracy of issues addressed in the report, the Region's
concurrence or nonconcurrence with the recommendations and plans for taking corrective action,
with milestone dates.
Please contact Howard Levin at 6-7522 to schedule an exit conference.
cc: Region 5 Senior Leadership Team
Howard Levin
Anna Miller
Recycled/Recyclable • Printed with Vegetable Oil Based Inks on 50% Recycled Paper (20% Postconsumer)
l he original response was signed by brands X. Lyons.
28
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
Appendix 1
Page 2 of 7
Region 5 Comments on Draft OIG Report
On Improving Region 5's EnPPA/PPG Program
Executive Summary
Page ii. Communication Key to Success - The first sentence should begin with "Region 5
striking " The EnPPA/PPG program in..." to include all participants in EnPPA negotiation and
management, to make this comment more inclusive. The Region's "program" includes all those
in media programs who are working with States under an EnPPA or PPG paradigm.
Page ii. first full paragraph - The Region does not agree with the statement "[a]s a result, two
states and one tribe that have participated in the program in past years have either dropped out or
are re-evaluating their continued participation." One state, Ohio, elected not to pursue an EnPPA
or PPG for FY 2000; however, the state advised the Region that the reason for not going forward
was that Ohio EPA's new management needed time to determine whether changes would be
made to the state's environmental priorities. Ohio did not inform U.S. EPA that it was dropping
out of the program due to communications problems with the Region. Ohio is the only state that
dropped out of the program this year.
Page iii- The Region believes that EnPPA development activities need to be accounted for other
than under specific media goals. We believe that media goals should contain the elements
required to manage them, as programs always have in the past. The issue is, we believe, not with
how EPA budgets to fund the administration of work on EnPPAs and PPGs. The issues raised
with GPRA have more to do with how States can or cannot account for funds along GPRA
subobjective lines, as is described elsewhere in the report. We do agree that the Region needs to
be able to quantify the resource expenditures on developing EnPPAs and PPGs.
Chapter 1
Page 3. first full paragraph, last sentence -Region 5 suggests revising this sentence to state "This
new partnership should also enable EPA and its partners to move beyond relying solely on
numbers of permits issued,...." This change is suggested to clarify that the system envisioned by
NEPPS includes a full spectrum of measures, including numbers of permits, as well as measures
reflecting changes in environmental quality.
Page 4. second paragraph after the table - The description of the Region's NEPPS structure fails
to include the expectation of Regional program roles beyond State teams and their members.
The role of program staff is described in the EnPPA/PPG Process (updated 6/99). The system
was designed to leave major programmatic negotiations and decisions in the programs, where the
expertise on environmental issues and program management remains.
29
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
Appendix 1
Page 3 of 7
Chapter 2
Page 7. first paragraph - Please omit "The EnPPA/PPG program in..." so that sentence begins
"Region 5 has not..." to include all participants in EnPPA negotiation and management to make
this comment more inclusive. The Region's "program" includes all those in media programs
who are working with States under an EnPPA or PPG paradigm.
We agree that the selected lack of team member authority to negotiate for their programs has
been a barrier. A corollary that should be reflected here is that there is sometimes a lack of
program staff participation in or contribution to the State team's work. In other words, it is a two
way street for information into and out of the State teams, and we need to improve the flow in
both directions.
As stated in earlier comments, the Region does not agree with the statement "[a]s a result, two
states and one tribe that have participated in the program in past years have either dropped out or
are re-evaluating their continued participation." It would be more accurate to say that one state
has withdrawn from an EnPPA/PPG and another may have considered withdrawing (although
the Region has not been informed of this). We also ask that the report recognize that there are,
as we have been told, several distinct reasons for the state's withdrawal, some of which have to
do with state agency management changes.
Page 8. first bullet -The Water Division notes that immediately following the identification of
quality assurance issues during the 1999 Ohio EnPPA process, the Region 5 EnPPA and PPG
lead staff worked with all parties involved to resolve the issues. The Region included a term and
condition as part of the FY 1999 PPG to address quality assurance issues with the Clean Water
Act Section 106 program and initiated a process to improve communications among programs
and states.
Page 8. second bullet -The report should include which Illinois Environmental Protection
Agency (IEPA) PPG application was reviewed and used as the basis for the information
presented. Based on the Executive Summary, it is assumed it was the FY 1999 application. The
report should include additional information to clarify that the "over allocation of water program
funds" in the IEPA PPG application was solely related to the Public Water Supply system
(PWSS) program. IEPA's application was $16,500 above the allotment from U.S. EPA
headquarters. This was not a significant discrepancy, and the Region believes the PPG award
could have been initiated without receipt of a revised application from IEPA.
Page 9. first bullet -The Region (Water Division and RMD) conducted numerous discussions
with the Fond du Lac Tribe regarding its QAPP, so it is unclear why the draft report states there
were "18 months with no feedback." The QAPP was not approvable as submitted because it was
based on a model for a Superfund site, not for water-related investigations; there were therefore
many issues for the Tribe to address to make the QAPP approvable.
2
30
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
Appendix 1
Page 4 of 7
Page 9. discussion of barriers -Region 5 notes that an additional reason why oversight and
reporting requirements may not have been reduced as significantly as states expected as a result
of NEPPS is that many Region 5 programs had already reduced reporting requirements prior to
NEPPS. A joint Region 5-Illinois EPA project that resulted in a chart of all programs' reporting
requirements and their source demonstrated that significant reporting reductions had already
been achieved. Most remaining requirements are required by statute, regulation or national
reporting systems.
Page 9. last paragraph and page 10 - Only one Region 5 state, Ohio, has dropped out of the PPG
program.
Page 10. first partial sentence - Please clarify that the "internal management changes" were
within the Ohio EPA.
Page 10. Conclusion -An additional reason why some of the benefits EPA and states expected
from PPGs have not been realized is that there have not been many requests from states to shift
funds in a PPG from traditional base program activities.
Recommendation 2-1— The Region concurs with the recommendation and has, since the events
cited occurred, taken this step to improve the quality assurance process. (It is assumed this
recommendation refers to Quality Management Plans (QMPs), since QAPPs are no longer
required.) The review role for multi-media QMPs is shared between the Resources Management
Division (RMD) and program offices, with the RMD serving as the lead for the reviews through
the Quality Assurance (QA) Core. Program offices provide input through their division
representatives to the Regional QA Core. The Regional QA Core established a "Procedure for
Review and Approval of All State Agency Quality Management Plans (QMPs)" on March 5,
1999. In May 1999 the Water Division developed supplemental procedures ("Steps in the QMP
Review Process") to use internally in reviewing state-submitted QMPs. These procedures are
being updated and will be finalized by March 31, 2000.
Regarding the QAPP for the Fond du Lac Tribe, the Quality Assurance Manager for the Water
Division will work with all interested parties to conduct an in-depth review of the events that
occurred at the time the Tribe submitted its QAPP for approval. The QA Manager will inform
the Water Division Director of suggested modifications to the Region 5 and Water Division
review procedures by March 31, 2000.
The current reorganization of the Water Division should also improve the timeliness of
communication and identification and resolution of QA issues.
Recommendation 2-2-The Region concurs with the need to ensure timely elevation of concerns
to the Region 5 Senior Leadership Team. The document entitled "Region 5 EnPPA/PPG
Process". Attachment II establishes roles and responsibilities for the EnPPA/PPG process,
3
31
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
Appendix 1
Page 5 of 7
including elevation of issues to SLT from the EnPPA Team Lead through a state's sponsoring
SLT lead. The Region will continue to stress the need for coordination and elevation of issues
that could affect adversely an entire EnPPA/PPG.
Recommendation 2-3-The Region concurs with this recommendation and will amend the
"Region 5 EnPPA/PPG Process" by March 31, 2000 to include a discussion of the need to
continue discussions with a state during the period between finalization of the EnPPA and
payment of the grant. In addition, the State/EPA Grants Workgroup has discussed this issue and
the award process and is developing a regional policy to ensure states receive a major portion of
their allotments as soon as possible.
Recommendation 2-4-Region 5 concurs with this recommendation and notes it held a meeting
with participating NEPPS states in November 1998 to discuss best practices and share other
information. The Region will hold another meeting during 2000 where we will discuss best
practices and development of environmental outcome measures The Region also gathers best
practices information through participation at national meetings and through national
workgroups.
Chapter 3
Page 12. first paragraph - Since measures have different origins, national or regional, Region 5's
ability to change and improve the measures varies in degree. Therefore we ask that the report be
more specific in referring to measures. Please clarify which measures are referred to by the
statement "...the quality of measures in the workplan varied." Does it refer to national core
performance measures, regional measures, perhaps State generated measures or all measures?
Page 13. Better Measures Can Improve Accountability - We believe the report should clarify the
difference in the numbers of measures between the two States to avoid erroneous conclusions.
For instance, the statements here imply mistakenly that the Region requires different numbers of
measures from different States. It is not immediately obvious that the difference in number of
measures may be due to the use the State has for the EnPPA. For example, Minnesota PCA
attempts to use its EnPPA to satisfy certain State reporting requirements that are the source of
many of the 537 measures.
Conclusion-The Region notes the conclusion discusses both accountability and managing for
environmental results. We interpret the conclusion to mean that in order to fulfill the
expectations of NEPPS, the Region and states need to develop environmental outcome measures
that meet the four criteria (time specific, quantifiable, measurable and verifiable) and thus
provide sufficient accountability. These measures will not, however, totally replace output
measures. The U.S. EPA's view is that a mix of outputs and environmental measures is
necessary to demonstrate accountability under NEPPS; while outputs alone might demonstrate
sufficient accountability under a system other than NEPPS, the expectation of NEPPS is that
improved environmental results are an important-but not the only-measure of accountability.
4
32
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
Appendix 1
Page 6 of 7
Recommendation 3-1-The Region concurs with this recommendation to the extent it applies to
measures developed jointly by the Region and states. It is important to recognize that U.S. EPA
headquarters program offices require EnPPAs to include core performance measures (CPMs),
unless a state demonstrates a compelling reason not to report on a CPM. Headquarters develops
CPMs, and the Region has little control over whether they are time specific, measurable,
quantifiable and verifiable. For other measures developed jointly by the Region and a state, the
Region will endeavor to ensure measures meet these four criteria. The Region will also amend
the Region 5 EnPPA/PPG Process" document by March 31, 2000 to emphasize the need to
evaluate whether proposed measure meet the four criteria.
In addition, the Region and State Agencies have had several discussions on the "burden
reduction" issue which is somewhat related to the nature of measures. Work on this topic, which
will be discussed at the May 2000 State Directors Meeting, will also help towards developing
appropriate measures that meet the four criteria.
The recommendation should be changed, however, to encompass all responsible parties by
stating "Require regional EnPPA team members and appropriate program staff to review
performance measures during negotiations..."
Recommendation 3-2 —The Region concurs with this recommendation.
Recommendation 3-3 -The Region agrees that development of outcome measures is a priority.
The Region plans to address this topic in its next meeting with all Region 5 NEPPS states. The
Region 5 NEPPS workgroup will also undertake a project to find good examples of outcome
measures to present at this meeting.
Chapter 4
Page 17. Second Paragraph - This paragraph states that limited resources are focused on tribal
grant issues, but concludes that the post-award policy provides for visits to the States every two
years. This discussion should focus on how the policy will be applied to the Tribes.
Recommendation 4-1-The Region agrees with the need to ensure a common understanding
among programs about accountability above and beyond the standards set forth in grant
regulations. The different requirements of different programs makes it difficult, however, to
develop regional guidance on minimum standards of accountability. The Region 5 NEPPS
Workgroup will hold informational sessions with each program office before August 31, 2000 to
encourage a common approach to determining whether measures (and EnPPAs) provide a
minimum standard of accountability.
Recommendation 4-2—The Region concurs with this recommendation.
5
33
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
Appendix 1
Page 7 of 7
Recommendation 4-3—The Region concurs with this recommendation.
Chapter 5
Page 20. last Paragraph - The report should clarify that there are two issues being identified (1)
that Regional work on EnPPAs/PPGs may not be reflected well in the Results Act goals, and (2)
that where funds given to States are concerned, Results Act reporting is based on budgeted
amounts not expenditures, due to the nature of the PPG approach whereby States are not
required to report on the same activity level.
Recommendation 5-1 - The Region will continue to work with OCFO and participate on the
national Planning Network to assess and improve the Agency's budget and accounting system,
but we do not agree that EnPPA/PPG work must be placed any differently within the GPRA
goals of U.S. EPA. The current direction within the Agency to place resources within the
environmental goals they support as much as possible. For example, there was a recent exercise
to move part of the Goal 10 resources back into the environmental Goals which they supported.
Region 5 believes it is appropriate for regional resources used in the development of EnPP A
agreements to be found under the various environmental goals as our efforts clearly will impact
the environment through the implementation of these agreements. We see the HQ resources for
NEPPS under Goal 7 to more likely be considered administrative resources to support the overall
NEPPS program, without directly impacting any one agreement, thus being at least somewhat
more removed from environmental progress.
Recommendation 5-2 - Region 5 does agree that it is useful to have a mechanism to measure the
resources associated with using NEPPS as a tool to further environmental progress. Therefore,
the Region will implement periodic assessment of the amount of time people spend working on
aspects of the NEPPS program ( This means negotiation, processing, follow-up; program
implementation is exempted because it is carried out regardless of NEPPS). This will allow us to
develop some sense of the resource investment NEPPS requires as a tool, as compared to other
grant processes, and whether process improvements can be made such that the NEPPS process
becomes more streamlined and efficient and yet still achieves the same or better environmental
results.
Recommendation 5-3 - The Region agrees with this recommendation, but may need to conform
to nationally-made decisions on portraying Results Act information.
34
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
Appendix 2
Page 1 of 2
DISTRIBUTION
Headquarters
Agency Follow-up Official (3101), Attn: Assistant Administrator for Administration and
Resource Management
Agency Follow-up Coordinator (3304), Attn: Director, Resource Management Division
Deputy Associate Administrator for Office of State and Local Relations (1306)
Headquarters Library (3404)
Region 5
Regional Administrator (R-19J)
Senior Leadership Team
Audit Follow-up Coordinator (MFA-10J)
Comptroller (MF-10J)
Public Affairs (P-19J)
NEPPS Workgroup
Library (PL-12J)
Region 5 States
Director, Illinois Environmental Protection Agency
Environmental Policy Advisor, Illinois Environmental Protection Agency
Commissioner, Indiana Department of Environmental Management
Director, Office of Planning and Assessment, Indiana Department of Environmental
Management
Director, Michigan Department of Environmental Quality
Deputy Director, Michigan Department of Environmental Quality
Commissioner, Minnesota Pollution Control Agency
PPA Coordinators, Minnesota Pollution Control Agency
Director, Ohio Environmental Protection Agency
Program Liaison, Ohio Environmental Protection Agency
Secretary, Wisconsin Department of Natural Resources
PPA Coordinator, Wisconsin Department of Natural Resources
35
Report 2000-P-00008
-------
Improving Region 5's EnPPA/PPG Program
Appendix 2
Page 2 of 2
Region 5 Tribe
Tribal Chair, Fond du Lac
Director, Division of Resources Management, Fond du Lac
Director, Environmental Services, Fond du Lac
Office of Inspector General
Inspector General (2410)
Office of Planning, Analysis and Results (2421)
Deputy Assistant Inspector General for External Audits (2421)
Acquisition and Assistance Audit Staff (2421)
36
Report 2000-P-00008
------- |