s A i
I vSv f
\ 0* OFFICE OF INSPECTOR GENERAL
PRC>1feC^
Catalyst for Improving the Environment
Audit Report
Stronger Leadership Needed to
Develop Environmental Measures for
Clean Water State Revolving Fund
Report No. 2004-P-00022
June 23, 2004

-------
Report Contributors:
Andres Calderon
Randy Holthaus
Richard Howard
Abbreviations
BOD	Biochemical Oxygen Demand
CWSRF	Clean Water State Revolving Fund
DO	Dissolved Oxygen
EPA	U.S. Environmental Protection Agency
GAO	General Accounting Office
OMB	Office of Management and Budget
Cover photo: Sable Falls, Grand Marais, Michigan (from EPA web site)

-------
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
OFFICE OF
INSPECTOR GENERAL
June 23, 2004
MEMORANDUM
SUBJECT:	Final Report: 2004-P-00022
Stronger Leadership Needed to Develop Environmental Measures for
Clean Water State Revolving Fund
FROM:	Tftic&aet ft. IRcc&eif,
Michael A. Rickey
Director for Assistance Agreement Audits
TO:	Benjamin Grumbles
Acting Assistant Administrator for Water
This is our final report on the subject audit conducted by the Office of Inspector General (OIG)
of the U.S. Environmental Protection Agency (EPA). This report contains a finding that
describes problems the OIG has identified and corrective actions the OIG recommends. We
discussed our finding with your staff and issued a draft report. We have summarized your
comments in this final report and included your complete response in Appendix D. This report
represents the opinion of the OIG and the findings do not necessarily represent the final EPA
position. Final determinations on matters in this report will be made by EPA managers in
accordance with established audit resolution procedures.
Action Required
In accordance with EPA Manual 2750, you are required to provide a written response to this
report within 90 calendar days of the date of this report. Please e-mail an electronic version of
your response to holthaus.randv@epa.gov. You should include a corrective actions plan for
agreed upon actions, including milestone dates. We have no objections to the further release of
this report to the public. For your convenience, this report will be available at
http://www.epa.gov/oig.
We want to express our appreciation for the cooperation and support from your staff during
this audit. If you or your staff have any questions about this report, please contact me at
(312) 886-3037, or Randy Holthaus, Assignment Manager, at (214) 665-6620.

-------

-------
Executive Summary
Purpose
As of 2003, the Clean Water State Revolving Fund (CWSRF) had $47 billion
available for projects. Given the dollars involved and the current need to show
the public the environmental benefits achieved from dollars invested, we reviewed
U.S. Environmental Protection Agency (EPA) and State efforts to measure the
environmental results of the CWSRF. Our specific audit objectives were to
answer the following questions:
What plan does EPA have to ensure that the environmental value of the
CWSRF can be measured?
What efforts has EPA made to measure the environmental results of CWSRF
projects?
What actions have States taken to measure CWSRF results?
Results in Brief
EPA needs to increase its leadership role in measuring the environmental benefits
of the CWSRF. EPA has been working on developing environmental measures
since 1998. However, EPA and the States have not established a uniform set of
measures to assess the environmental impact of the program. Further, EPA has
not developed a comprehensive plan for measuring the results of the CWSRF. As
a result, EPA: (1) did not know the actual environmental impact of the CWSRF
and will not know unless it develops measures; and (2) cannot compare the impact
of individual water quality programs and make informed resource allocations.
Also, some States questioned the value of measuring and, therefore, did not place
emphasis on doing it.
Recommendation
We recommend that the Assistant Administrator for Water develop a plan, with
milestone dates, that (a) establishes the value of measuring environmental
benefits, (b) seeks input from other stakeholders about measuring options,
(c) identifies and evaluates measuring options, and (d) selects an option and
establishes an implementation plan.
i

-------
Agency Comments and OIG Evaluation
The Office of Water generally concurred with our finding and recommendation.
The Office of Water plans to pursue the ideas in the recommendation through
various workshops and workgroups. The main goal is to have a suite of proposed
indicators developed by February 2005. The Office of Water will develop a
performance measurement plan that will include activities with appropriate
milestone dates.
We agree with the Agency's proposed action. However, in its response to our
draft report, the Office of Water did not provide a specific milestone date for
when it will finalize its performance measurement plan. As a result, we are
requesting that the Office of Water provide a specific milestone date within
90 days of the final report, or a copy of that implementation plan if it is
completed.

-------
Table of C
Executive Summary	 i
Chapters
1	Introduction 	 1
Purpose	 1
Background	 1
Scope and Methodology	 4
2	CWSRF Leadership Needed	 5
Importance of Measuring for Results 		5
EPA Has Financial But Not Environmental Measures 		6
EPA Needs a Comprehensive Plan for Measuring
Environmental Results of the CWSRF		7
Conclusion 		8
Recommendation 		9
Agency Comments 		10
OIG Evaluation 		10
Appendices
A Details on Scope, Methodology, and Prior Audit Coverage	 11
B EPA's Efforts to Measure the CWSRF 	 13
C Summary of Measurement Activities for Selected States	 17
D EPA Response 	19
E Distribution 	23
iii

-------

-------
Chapter 1
Introduction
Purpose
Since 1988, the U.S. Environmental Protection Agency (EPA) has provided States
about $21 billion to capitalize the Clean Water State Revolving Fund (CWSRF).
State funds, interest income, principal repayments, and bond revenue provided
another $27 billion. As of 2003, the CWSRF had about $47 billion available for
projects.1
Success of the CWSRF program has been measured almost entirely based on
financial indicators. Given the dollars involved and the need to show the public
the environmental benefits achieved from dollars invested, we reviewed EPA's
and States' efforts to measure the environmental results of the CWSRF.
Our specific audit objectives were to answer the following questions:
•	What plan does EPA have to ensure that the environmental value of the
CWSRF can be measured?
•	What efforts has EPA made to measure the environmental results of CWSRF
projects?
•	What actions have States taken to measure CWSRF results?
Background
Clean Water Act and Creation of the CWSRF
During the 1970s and 1980s, the Construction Grants Program was a major source
of Federal funds, providing more than $60 billion for the construction and
rehabilitation of publicly-owned wastewater treatment facilities. Despite the
success of the Construction Grants Program, Congress sought to establish a more
sustainable method of financing the construction of wastewater treatment plants.
In 1987, Congress passed amendments to the Clean Water Act designed to phase
out the Construction Grants Program and shift municipal financial assistance from
grants to loans. As a result, the CWSRF began operating in fiscal year 1989, and
Congress designated 1990 as the last year that grant funds would be appropriated
for the Construction Grants Program. This new approach to funding water
' As of 2003, the CWSRF was valued at $48 billion, but about $1 billion was not available for projects
because $800 million was used to administer the Fund and another $300 million was transferred to the Drinking
Water State Revolving Fund.
1

-------
pollution abatement projects was designed to be a permanent, State-operated
financial assistance program. The CWSRF was charged with funding a wide
variety of water quality projects, including all types of non-point source,
watershed protection or restoration, and estuary management projects, as well as
more traditional municipal wastewater treatment projects (point sources).
How the CWSRF Program Works
Through the CWSRF program, all 50 States have a revolving loan fund that
provides independent and permanent sources of low-cost financing for a wide
range of water quality projects. Initially, EPA provides grants to States to
establish and further fund States' CWSRF programs; States are required to
provide matching funds (equal to 20 percent of the Federal grant amount).
Subsequently, the States run their CWSRF programs and make CWSRF loans,
primarily to communities. Loan repayments are recycled back into each
individual State's CWSRF program to fund new water quality projects within the
State.
Federal funds for the CWSRF are allocated to each State based on a formula in
the Clean Water Act. From 1989 to 2004, States received Federal funds ranging
from about $110 million to $2.48 billion. Using fund assets as collateral, some
States issue bonds to leverage their CWSRF programs (secure additional funding).
Such leveraging has added $14 billion to the CWSRF for water quality projects.
Loan repayments and interest earnings have added $9.1 billion.
As of the end of fiscal 2003, the total CWSRF funds available for projects was
about $47 billion. In 2003, Congress appropriated about $1.34 billion to EPA -
about one-sixth of its budget - so that the Agency could provide additional
CWSRF capitalization grants to States.
EPA's Role Under the CWSRF
EPA's Office of Wastewater Management, within the Office of Water, is
responsible for overseeing the CWSRF program on a broad level. The Office of
Wastewater Management establishes oversight policies for the CWSRF and
annually reviews how well the EPA regions monitor the program. Each of EPA's
10 regional offices has the responsibility of awarding, monitoring, and closing out
capitalization grants. Further, the regions are to:
Work with States to improve their CWSRF programs.
Ensure that States comply with eligibility requirements.
Encourage States to fund the highest priority problems.
Ensure that its States' CWSRF programs are financially stable and viable.
• Promote the implementation of national priorities.
2

-------
States' Roles Under the CWSRF
Under the Clean Water Act, the CWSRF program is State-managed and directed.
The States are ultimately responsible for selecting projects to receive loans. Each
State annually outlines how it plans to use all available funds during the year.
Each State also has its own specific procedures for reviewing potential projects,
such as financial hardship, relative water quality benefits, location within high-
priority watersheds, or other relevant factors. In addition, States are responsible
for setting the terms of the loans.
After States receive the capitalization grants from EPA, States make the loans to
communities, individuals, and other high-priority recipients. States can also form
partnerships with other funding sources - such as banks, local governments, and
State agencies - to extend credit for promoting water quality. In recent years,
States have begun to devote an increasing number of loans to non-point source
(e.g., development of stream bank buffer zones) and estuary management (e.g.,
restocking fish) projects. However, as of the end of fiscal 2003, 65 percent of the
total loans and 96 percent of the total dollars went toward point source projects
(see box). These point source projects were for replacement, upgrade, or
modification of inadequate or failing wastewater treatment systems, as well as
installation of new systems. The remaining 4 percent of the dollars went for
non-point source and estuary projects.
CWSRF
Program
Dollars Provided For Projects
Number of Loans
Nonpoint Source and Estuary
Nonpoint Source and Estuary
Si.8 Billion (4 percent)
35 percent
I Point Source 1
~~~ J
V $40.9 Billion /
\ Point Source j
\ (96 percent)/
\ 65 percent /
Source: EPA, Office of Water

Data through 2003

3

-------
Recent Federal Emphasis on Results
The Government Performance and Results Act of 1993 stresses that agencies
should identify program goals and performance measures and link them with the
budget process. In August 2001, the President announced an aggressive strategy
for improving management and performance of the Federal government. The
President's Management Agenda is guided by three principles, one being that
government should be "results-oriented." A current initiative flowing from this
principle is the integration of budget and performance.
The Office of Management and Budget (OMB), using an analytical tool called the
Program Assessment Rating Tool, is currently assessing the performance and
achievements of Federal agencies. OMB's review consists of a series of questions
that focus on four key areas. The rating tool provides a common, transparent
approach to assessing programs. In 2004, OMB's overall weighted score for the
CWSRF was 52 out of 100, with an overall rating of "Results Not Demonstrated."
EPA's individual raw scores for each of the four key areas were:

Score
Program Purpose and Design
80
Strategic Planning
25
Program Management
100
Program Results
27
OMB is allowing EPA time to improve its performance in this area, and expects
substantial progress by the next review cycle. OMB expects EPA to tie
environmental outcomes, such as number of stream miles no longer impaired,
directly to the CWSRF program.
Scope and Methodology
We performed our audit in accordance with the Government Auditing Standards,
issued by the Comptroller General of the United States. We conducted our audit
field work from June 2003 to January 2004. We conducted much of our field
work at EPA Headquarters, and gathered information on the CWSRF program
from 8 of EPA's 10 regions and from 11 States. See Appendix A for details on
the scope and methodology.
4

-------
Chapter 2
CWSRF Leadership Needed
EPA needs to increase its leadership role in measuring the environmental benefits
of the CWSRF. EPA has been working on developing environmental measures
since 1998. However, EPA and the States have not established a uniform set of
measures to assess the environmental impact of the program. Further, EPA has
not developed a comprehensive plan for measuring the results of the CWSRF. As
a result, EPA: (1) did not know the actual environmental impact of the CWSRF,
and will not know unless it develops measures; and (2) cannot compare the impact
of individual water quality programs and make informed resource allocations.
Also, some States questioned the value of measuring and, therefore, did not place
emphasis on doing it.
Importance of Measuring for Results
Ultimately, EPA must be able to answer four key questions (see box) to ensure
that environmental results are achieved in a cost-effective manner. EPA has built
a framework that aligns planning,
budgeting, analysis, and accountability
into an integrated system. By planning
strategically, measuring performance,
analyzing data, and using what was
learned, EPA can make sound decisions
about how to use its resources.
Measuring for results is a key process
within this system that involves
assessing progress and linking actual
resources used to the actual results
achieved.
The General Accounting Office (GAO), in its January 30, 2004, report,
Performance Budgeting: Observations on the Use of OMB 's Program Assessment
Rating Tool for the Fiscal Year 2004 Budget, noted that how grants are structured
plays a role in whether Federal agencies are able to hold third parties responsible
for results. Programs such as the CWSRF present implementation challenges,
especially in those instances in which national goals are not compatible with State
and local priorities. Many of the outcomes for which Federal programs are
responsible are part of a broader effort involving Federal, State, and local partners.
Therefore, it is often difficult to isolate a particular program's contribution to an
outcome. Further, evaluation data may be limited because of constraints on
Federal agencies' ability to influence program outcomes and reliance on States.
	
EPA must be able to answer the
following questions:
•	Did we accomplish what we
planned?
•	Did we keep within our budget?
•	Did we achieve the environmental
results we desired?
•	What did the Agency spend to
achieve those results?
"Managing for Results," Office of
the Chief Financial Officer web site
5

-------
In another GAO report, Managing for Results: Measuring Program Results that
are Under Limited Federal Control, dated December 11, 1998, GAO found that
Federal agencies should select a mix of performance goals that include
intermediate and end outcomes. This allows agencies to minimize the risk due to
limited control over external factors. In addition, because Federal agencies
sometimes find it difficult to confidently attribute a causal connection between
one of its programs and desired outcomes, GAO found that stakeholder
involvement is vitally important in the process of developing practical and
broadly-accepted performance measures.
EPA Has Financial But Not Environmental Measures
EPA has for years had some good indicators to measure the financial stability and
success of the CWSRF. EPA collects and reports annually on the financial
aspects of the CWSRF, including figures on loans made, projects started, interest
payments, and loan repayments. One financial performance indicator is return on
Federal investment, which shows how many dollars of assistance were disbursed
to eligible borrowers for each Federal dollar spent. Another indicator - fund
utilization rate - is designed to show how many dollars of assistance were in use
for each Federal dollar that could be loaned out.
EPA's Strategic Plan defines key goals in environmental and public health terms,
including the expected improvements in key measures by 2008. For the CWSRF,
the Strategic Plan states that, over the next 5 years, EPA will work to "link
projects to environmental results through the use of scientifically sound water
quality and public health data."
While the EPA Office of Water is making an effort to address environmental
results through its strategic planning process, EPA has not developed outcome
measures for the CWSRF. EPA has been working the last several years to
develop environmental measures. Specifically, EPA has:
Formed a workgroup with States to discuss how to measure benefits.
Reported on the national improvement to water quality since passage of the
Clean Water Act.
Initiated a pilot study on environmental measures.
Hired a contractor to help define environmental benefits.
Made grant funds available to demonstrate environmental benefits.
Encouraged States to report on environmental benefits in their annual reports
to EPA.
Details on each of these efforts are in Appendix B.
EPA officials noted that they need to find a cost effective way to gather project
level data. However, EPA does not have specific milestone dates for doing so.
6

-------
In spite of the efforts undertaken, EPA and the States have not established a
uniform set of environmental measures to assess the program's environmental
impact. Some possible measures have been identified but need to be further
researched and developed. For example, in the March 2001 pilot study, EPA and
States tested six potential environmental measures for CWSRF projects. Other
efforts have identified other possible measures, such as increased compliance with
National Pollutant Discharge Elimination System permits, and estimated
reduction in biochemical oxygen demand and increase in dissolved oxygen within
waterbodies. (See Appendix C for a summary of the measurement activities
conducted by States.)
Other efforts within EPA and the government can provide alternatives for
measuring CWSRF results. To date, most of the suggested measures have focused
on point source projects. There are other efforts within EPA and throughout the
government to measure the impact of non-point source projects. For example, in
2002, EPA began requiring recipients of Section 319 grants (non-point source
projects) to report loading reductions for nutrients and sediments. Grant
recipients must also report on acres of wetlands restored or created, and number of
feet of streambank protected and stabilized.
EPA Needs a Comprehensive Plan for Measuring
Environmental Results of the CWSRF
Although EPA's 2004 to 2008 Strategic Plan indicates that outcome measures will
be in place for improving water quality at the watershed level, EPA does not have
a clear plan as to how it will develop measures for the CWSRF. EPA's current
approach has been a reaction to criticism by OMB and others, and does not appear
to be part of a comprehensive, organized process.
EPA officials have identified activities that they believe will link CWSRF projects
to environmental results. However, EPA has not identified how or when the
activities will be implemented, or determined whether these activities will lead to
environmental measures for the CWSRF. To date, the Office of Water has not
answered questions such as:
•	How will CWSRF environmental results be integrated into planning and
budgeting decisions by EPA, the States, and Congress?
•	What measures are feasible?
•	How will the measurements be conducted and funded?
•	Who will be responsible for collecting, organizing, and analyzing the data?
The Office of Water needs to identify and evaluate alternatives for measuring
environmental benefits of the CWSRF program. GAO, in its report Managing for
Results: Measuring Program Results that are Under Limited Federal Control,
7

-------
suggests that, in situations where agencies believe they have limited control over
outcomes, agencies can: (1) select a mix of outcome goals over which the agency
has varying levels of control; (2) redefine the scope of a strategic goal to focus on
the more narrow range of their actual activities; (3) disaggregate goals for distinct
large populations for which the agency has different expectations; or (4) use data
on external factors to statistically adjust for their effect on the desired outcome.
Adopting a strategy that incorporates some or all of these methods should help
agencies minimize the risk, due to their limited control over external factors.
According to GAO, if unexpected events prevent agencies from achieving their
end outcome, they may be able to demonstrate their effectiveness through an
intermediate outcome.
Another key step in developing environmental measures for the CWSRF is to
establish the value of the measures for State partners and define how the
measurement data will be integrated into planning and budgeting decisions.
A December 2003 report by the IBM Center for the Business of Government -
Strategies for Using State Information: Measuring and Improving Program
Performance - emphasized that, when creating measures, it is vitally important to
identify how data supporting those measures will ultimately be used. This report
states that Federal agencies had greater success in establishing performance
measures when they emphasized the value of the information in making decisions.
Federal agencies should strive to collect, organize, and make performance
information available for use by others.
Because EPA has not identified how results will be integrated in planning and
budgeting decisions, the States have raised concerns about investing in measuring
results for the CWSRF program. According to the IBM Center report, to lessen
the chances that States will attempt to dismantle the measurement system, Federal
agencies should make it a priority to build measurement systems that serve the
needs of all stakeholders whose actions contribute to improved outcomes and who
face choices among options that might be influenced by the performance
information.
Conclusion
EPA has used a short-term, reactive approach for identifying ways to measure
environmental benefits of the CWSRF. EPA has some general knowledge of the
impact that wastewater infrastructure projects have had on water quality.
However, EPA does not specifically know what contribution the CWSRF has
played in this improvement. Consequently, EPA does not know what the actual
environmental impact of the CWSRF is, and will not know in the future unless it
develops measures. Further, EPA cannot compare the impact of individual water
quality programs and make informed resource allocations.
8

-------
The CWSRF is just one of six
strategies (see box) that EPA is using
to achieve improved water quality.
EPA needs to be able to identify the
contribution that each of its
strategies is making towards
improving water quality. If EPA is
unable to do this, EPA has no way of
analyzing the effectiveness of the
program in improving water quality
as compared to other strategies or
tools. Analyzing results information
is an important component of EPA's
planning, budgeting, analysis, and accountability framework.
EPA must lead States in developing measures for the CWSRF. EPA needs to
decide how measures will be used in decision making by all stakeholders - EPA,
States, OMB, and Congress. EPA needs to seek out information from other
Federal agencies, research communities, and academia on methods for measuring
environmental results of CWSRF activities. EPA needs to identify what
information is currently available - within EPA, at the State level, and at the local
level. Further, EPA needs to analyze options for uniformly measuring results and
then select the option that will best meet the needs of all stakeholders.
Throughout this process, EPA needs to work closely with the States, which will
be the primary sources and beneficiaries of the measurement information.
Recommendation
2-1 We recommend that the Assistant Administrator for Water develop a plan,
with milestone dates, that:
Establishes the value of measuring environmental benefits by
identifying how this information would be used by EPA and States in
making future decisions about the CWSRF program.
Seeks input from other EPA offices, Federal agencies, States, and
other stakeholders on options for measuring environmental
contributions of the CWSRF program.
Identifies and evaluates options for measuring environmental benefits
and considers for each option: strengths and weaknesses;
feasibility of implementation by all States; cost; and validity of
available data.
Selects an option and establishes an implementation plan.
	
Strategies for Water Quality
•	Strengthen the Water Quality
Standards Program
•	Improve Water Quality Monitoring
•	Develop Effective Watershed
Plans and TMDLs
•	Control Nonpoint Source Pollution
•	Strengthen NPDES Permit
Program and Implement National
Industrial Regulation Strategy
•	Support Sustainable Wastewater
Infrastructure
9

-------
Agency Comments
Office of Water stated it generally agreed with the finding and recommendation,
and that our report reflects activities the office has already initiated, or is planning
to initiate, in the next 12 months. The Office of Water plans to pursue the ideas in
the recommendation through various workshops and workgroups. The main goal
is to have a suite of proposed indicators developed by February 2005. To address
the recommendation, the Office of Water will develop a performance
measurement plan that will include activities with appropriate milestone dates.
They believe the prospects for success are significantly improved by collaborating
with the States directly in the process of developing measures.
Office of Water officials believe the environmental impact of the program is
known, although they agreed that more precise and targeted measures are
necessary. The CWSRF program and the previous wastewater construction grants
program have played and continue to play a vital role in achieving and
maintaining compliance with water permits. Office of Water officials are
confident that because State priority systems help direct funding to
environmentally worthy projects, projects funded by the CWSRF are appropriately
results-oriented.
OIG Evaluation
We agree with the Office of Water's decision to develop a performance
measurement plan. However, in responding to the draft report, Office of Water
did not provide a specific date for when it would finalize the implementation plan.
Therefore, when responding to the final report, the Office of Water needs to
provide a specific date for completing the plan, or a copy of the plan if it is
completed.
10

-------
Appendix A
Details on Scope, Methodology,
and Prior Audit Coverage
Scope and Methodology
We conducted our work primarily at EPA Headquarters in Washington, DC.
We interviewed EPA Office of Water officials, including management and staff of
the CWSRF program, regarding efforts to establish environmental measures. We
discussed with officials their plans for future actions to improve their ability to
measure results in the future. We also reviewed EPA's Strategic Plan, in
particular the section on Goal 2 for Clean and Safe Water. We reviewed several
key EPA reports issued within the last 4 years related to water quality and the
CWSRF. We also reviewed minutes of 13 meetings that the State/EPA State
Revolving Fund Workgroup held from 1998 through 2003.
We reviewed 8 of EPA's 10 regions; we did not review Region 7 or Region 9.
The 8 regions we reviewed oversee environmental programs in 42 States. We
interviewed the CWSRF coordinators in those eight regions to determine (1) what
the regions were doing to encourage or assist States in measuring environmental
results, and (2) whether the regions were aware of what their States were doing or
had done in this area. We reviewed the State evaluation reports prepared by seven
of the eight regions to determine whether EPA regions discussed measuring
CWSRF for environmental results.
We interviewed State officials from Colorado, Delaware, Georgia, Illinois,
Massachusetts, New Jersey, Ohio, Oklahoma, Texas, Utah, and Washington. We
also visited the Ohio Environmental Protection Agency in Columbus, Ohio; and
the Delaware Department of Natural Resources and Environmental Control in
Dover, Delaware. For each of the 11 States, we asked CWSRF program managers
to describe their efforts to measure the impact of CWSRF projects. We also
inquired if there was a plan to measure results in the future and, if so, how and
when it would be implemented.
We chose the 11 States for several reasons. EPA identified Delaware as being on
the forefront of the measuring issue. We selected New Jersey, Ohio, Texas, and
Utah primarily because they took part in the 2001 EPA/State pilot study. We
interviewed officials from the remaining six States because they generally
included more environmental benefits information in their annual reports. We
intentionally selected States from different EPA regions to obtain a broad
perspective of State experiences and water quality challenges.
11

-------
We reviewed the most recent CWSRF annual reports for 49 of the 50 States -
30 from 2002 and 19 from 2003 - to determine what States reported on
environmental benefits. (We were unable to obtain the annual report for
Kentucky.)
We also spoke with officials at the National Academy of Public Administration
and GAO to determine whether any studies had been conducted on the cost of
measuring the benefits of Federal programs. We also reviewed several GAO
reports related to performance measurement in the Federal government, including:
•	Performance Budgeting: Observations on the Use of OMB 's Program
Assessment Rating Tool for the Fiscal Year 2004 Budget, GAO-04-174,
January 30, 2004
•	Management for Results: Measuring Program Results that are Under
Limited Federal Control, GAO/GGD-99-16, December 11, 1998
We spoke with OMB officials throughout the course of the audit to obtain their
perspectives on EPA's management of the CWSRF. We also reviewed the results
of OMB's application of the Program Assessment Rating Tool to CWSRF.
We conducted our work from June 2003 to January 2004. We performed the
audit in accordance with Government Auditing Standards, issued by the
Comptroller General of the United States.
We issued the draft report to the Acting Assistant Administrator for Water on
April 26, 2004. The Acting Assistant Administrator responded on May 27, 2004.
An exit conference was held on June 22, 2004. In its response, Office of Water
provided comments to clarify portions of the report, and we incorporated that
information as appropriate. At the end of Chapter 2 we summarized Office of
Water's comments, and provided our evaluation of the comments. The full text of
the response is in Appendix D.
Prior Audit Coverage
We have not issued any other audit reports on programmatic or performance
issues related to EPA's measurement of CWSRF environmental benefits.
12

-------
Appendix B
EPA's Efforts to Measure the CWSRF
EPA has been working the last several years to try and develop environmental benefit measures
for the CWSRF but has made little progress. The following paragraphs describe the details on
each of the six bullets noted in Chapter 2.
Workgroup Activities
In 1998, EPA and the States formed a workgroup that meets twice a year to
discuss CWSRF regulations and issues, including environmental measures. In
November 2002, the workgroup formed a subgroup to address how to measure
environmental benefits of the CWSRF. Subgroup discussions have found the
following obstacles to measuring benefits:
•	Technical difficulties in attributing benefits specifically to the CWSRF,
especially when many other projects are affecting the same watershed.
•	Costliness of measuring.
•	Lack of data.
•	Differences among States - data from one State does not mean the same to
another.
•	Lack of uniform, widely-accepted environmental measures.
•	Questions about what use any data would have, other than to generate
statistics that would be meaningless to other States.
•	States' desire to invest funds in other projects rather than measuring.
•	Belief among several States that the up-front review process for loan
applications ensures their projects provide environmental benefits.
Despite these obstacles, the subgroup has succeeded in getting States to share
ideas and work together on a solution for measuring benefits. Some States have
cited the subgroup as a good forum for discussion, brainstorming, and networking.
Water Quality Report
In June 2000, EPA published a report, Progress in Water Quality - An Evaluation
of the National Investment in Municipal Wastewater Treatment. This report
stated that two key water quality indicators - biochemical oxygen demand (BOD)
and dissolved oxygen (DO) - improved significantly from 1968 to 1996 in some
key, large water basins. The importance of BOD and DO are:
13

-------
•	BOD is a measure of the oxygen-consuming organic matter and ammonia-
nitrogen in wastewater. The higher the BOD loading, the greater the depletion
of oxygen in the waterway, and the worse the water quality. When oxygen
becomes depleted from the waterway, the water becomes unhealthy to support
aquatic life.
•	DO is critical in the decomposition of organic carbon and organic nitrogen and
ammonia from wastewater discharges. Historical DO records provide an
excellent environmental measure for characterizing water quality responses to
long-term changes in wastewater loading. The higher the DO levels, the
higher the water quality.
According to this EPA study, despite a 35-percent increase in pollutant loadings,
the amount of BOD actually declined by 23 percent. This illustrates that the
investments in municipal wastewater treatment have resulted in dramatic
improvements in restoring water quality and biological resources while creating
thriving water-based recreational uses.
In a July 2003 report to Congress, Paying for Water Quality: Managing Funding
Programs to Achieve the Greatest Environmental Benefit, EPA noted that the
study on BOD and DO helps illustrate that modeling can be used to demonstrate
the benefits of clean water investments and successful projects, and for
determining compliance outcomes on a national basis. This report further stated
that EPA is working to enhance its available water quality modeling capabilities.
Pilot Study
In March 2001, EPA published a study, Development, Selection, and Pilot
Demonstration of Preliminary Environmental Indicators for the CWSRF
Program. In this study, six States - California, Michigan, New Jersey, Ohio,
Texas, and Utah - evaluated a preliminary set of environmental indicators
developed for the CWSRF. Each State reviewed a subset of their CWSRF
projects to determine whether it could measure, or otherwise reflect,
environmental improvement as a result of the projects. The study evaluated the
following six environmental measures:
•	Number of pounds of pollutants removed from the environment.
•	Number of pounds of pollutants prevented from entering the environment.
•	Physical changes to the terrestrial, riparian, or aquatic habitat and hydrology.
•	Waterbodies previously impaired, now improved or meeting designated uses.
•	Waterbodies protected.
•	Reduced health risks and/or increased recreational use.
14

-------
In addition, the study identified barriers to using the aforementioned measures:
•	Identifying and accessing environmental data related to CWSRF projects
ranged from problematic to difficult.
•	Applying environmental measures to projects other than wastewater treatment
upgrades or expansions was difficult.
•	Substantial modeling would be necessary to determine pounds of pollution
prevented from entering the environment; this modeling would be costly,
controversial, time-consuming, and only reflect estimated improvements.
•	Currently, project level environmental data is not tracked by most States.
•	States do not have procedures in place to collect information on environmental
outcomes.
Based on the findings of pilot testing, the study recommended how to proceed
with the development and application of environmental measures for the CWSRF.
Among other things, the report concluded that implementation of the measures be
at State's discretion, and that guidance material be developed to assist with
implementation.
Contract to Define Environmental Benefits
Beginning in January 2004, EPA authorized a contractor to perform a study on
how to measure the environmental benefits of the CWSRF. EPA and the
contractor will visit Georgia, Oklahoma, Texas, New York, Massachusetts,
Arizona, California, Hawaii, and Nevada. The purpose of these visits is to
determine if these States can measure the environmental benefits of the CWSRF,
and if anything being done in these States can be adopted by other States.
Grant to Demonstrate Environmental Benefits
In December 2003, EPA announced that grants funds were available under the
Clean Water Act, Section 104(b)(3), to demonstrate environmental benefits from
CWSRF projects. According to the EPA CWSRF Branch Chief, two applicants
have applied for grants. One applicant proposed to expand its project tracking
system to include benefits-related measures and information. The second
applicant proposed to hold workshops with interested States to develop voluntary
metrics.
States to Report on Environmental Benefits
Although there is no requirement for States to include CWSRF environmental
benefits in their annual report, EPA has encouraged States to do so. CWSRF
15

-------
coordinators in four of the eight EPA regions we spoke with encouraged their
respective States to report on environmental benefits.
About 70 percent of the State annual reports did not include environmental
information. The reports essentially included only financial data. Further,
20 percent expressed environmental benefits in very general terms. Only about
10 percent of the reports provided a somewhat detailed description of
environmental benefits, including some numerical data, generated by projects,
groups of projects, or the program. For those States that did report some
environmental information, most of the information was based on pre-
construction estimates of future benefits rather than actual benefits.
16

-------
Appendix C
Summary of Measurement Activities for
Selected States
Most States were not measuring the environmental results of CWSRF projects. States generally
were not measuring because: (1) EPA does not provide States any additional funding to measure
results, (2) it is extremely difficult to distinguish the benefits of the CWSRF from other projects
affecting a watershed, and (3) some States believe that the CWSRF is a financial program and
measuring for environmental impact does not fall within the scope of the program.
The following descriptions offer "snapshots" of what three States said they are doing to measure
the environmental impact of the CWSRF program. This appendix is not meant to be an inclusive
list. To identify what States were doing, we interviewed officials from 11 States, and reviewed
annual reports from 49 of 50 States (we were unable to obtain the annual report for Kentucky).
Because we did not interview all 50 States and annual reports can vary in content, it is possible
that there are other States that are performing environmental measurement efforts.
Colorado
Colorado arranged its projects by watershed in its 2002 Annual Report and listed
potential environmental measures of projects as well as examples of benefits.
The State believes that there is potential in the future to effectively measure the
environmental benefits of the CWSRF. Colorado identifies the potential
environmental benefits of CWSRF projects by coordinating efforts with other
program staff, including program staff that develops the Clean Water Act 303(d)
list. In 2004, Colorado is requiring systems to identify the potential
environmental benefits of the proposed projects in its annual Intended Use Plan
Survey. State staff are in the process of revising the loan application to request
information from applicants about the potential environmental impacts.
Delaware
Delaware engineers use a mathematical formula to determine how much
phosphorous is prevented from entering the environment by eliminating septic
systems and building wastewater collection systems (point-source issue). For
example, Delaware's Annual Report for 2003 states that Sussex County's project
for the Miller Creek Sanitary Sewer District will "eliminate 483 septic systems
and prevent the installation of 684 septic systems," resulting in the prevention of
"an estimated 20,837 pounds of nitrogen and 1,592 pounds of phosphorous from
reaching groundwater annually." Delaware also uses a formula to determine how
17

-------
much nitrogen is prevented from entering the environment by building roofed
poultry storage sheds (non-point-source issue).
Georgia
Georgia developed a process to capture project benefits - human health,
environmental, economic, and financial - resulting from all projects (Federal and
State) funded in 2003 and beyond. The benefits are captured for each project
funded and loaded into a web-based database, and are to be analyzed for reporting
within the 2004 CWSRF annual report to EPA and Georgia's State legislature.
It is Georgia's intention to report on the performance benefits of those projects
funded with CWSRF funds in 2003 and 2004. In addition, Georgia officials
established a forum to discuss new performance measures that are identified, how
those new measures could be added into the process, and how to continue to
improve the measures that are currently being captured.
18

-------
EPA Response
Appendix D
May 27, 2004
MEMORANDUM
SUBJECT: Draft OIG Audit Report: Stronger Leadership Needed to Develop Environmental
Measures for Clean Water State Revolving Fund, Assignment No. 2003-1002
FROM: Benjamin H. Grumbles/s/
Acting Assistant Administrator
TO:	Michael A. Rickey, Director
Assistance Agreement Audits
Thank you for the opportunity to review and comment on the draft report of the Office of
Inspector General, entitled: "Stronger Leadership Needed to Develop Environmental Measures
for the Clean Water State Revolving Fund."
The report makes the following recommendation (p.9):
We recommend that the Assistant Administrator for Water develop a plan, with milestone
dates, that:
Establishes the value of measuring environmental benefits by identifying how this
information would be used by EPA and States in making future decisions about
the CWSRF program.
Seeks input from other EPA offices, Federal agencies, States, and other
stakeholders on options for measuring environmental contributions of the CWSRF
program.
Identifies and evaluates options for measuring environmental benefits and
considers for each option: strengths and weaknesses; feasibility of implementation
by all States; cost; and validity of available data.
Selects an option and establishes an implementation plan.
We generally concur with these points and believe that for the most part they fairly reflect
the steps that we have already initiated or are planning to do within the next 12 months. Thus
far, my staff has visited with seven States (GA, TX, OK, CA, HI, NV, and AZ) and will soon
initiate a series of workgroup meetings under the aegis of the Association of State and Interstate
Water Pollution Control Administrators (ASIWPCA). We have found from our initial visits that
the States share our interest in the importance in improving the documentation of environmental
19

-------
results of the CWSRF program. We have already gained useful insights into how this might be
accomplished within existing resource and data constraints. We will pursue these and other ideas
through the ASIWPCA workshops, the State/EPA SRF Workgroup, and the annual SRF Training
Workshop in November 2004. Our main milestone is to have a suite of proposed indicators
developed by January-February 2005.
In an effort to address the recommendation in the draft report, we will develop a
performance measurement plan that will include the above activities with the appropriate
milestones. As with any plan, it will be modified to accommodate changed circumstance and
take advantage of new ideas and opportunities. The prospects for success are significantly
improved with the collaboration of the States directly in the process of developing the measures.
Their buy-in is essential to our collective success.
Specifically, I would like to clarify several matters raised in the draft report.
1)	At page 5, first paragraph, the draft report states that: "EPA and the States have not been
able to agree upon specific measures to determine the environmental impact of the program."
This statement leaves the misleading impression that we are in disagreement with the States over
measures. This is not the case, and for the sake of clarity, I would urge that the final report
reflect that fact. In 2001, EPA and several States formed an innovative task force that produced a
suite of seven environmental indicators. The recommendation of the group was that they be
considered optional essentially because of resource and monitoring issues. This initial effort was
an important collaborative step.
2)	Same paragraph, the draft report states that: "EPA did not know that the actual
environmental impact of the CWSRF." While we agree that more precise and targeted measures
are necessary, at the same time, we disagree that the environmental impact of the program is
unknown. Over 94 percent of CWSRF funding assistance goes to publicly-owned treatment
works that are permitted under the National Pollutant Discharge Elimination System (NPDES)
program. These permits are based on water quality standards established by the States for
receiving waters. The standards define beneficial uses for the receiving waters which the
permitted discharges are intended to protect or restore. We know that the CWSRF program and
its predecessor wastewater construction grants program have played and continue to play a vital
role in achieving and maintaining compliance with enforceable requirements of the Act
embodied in water quality standards and NPDES permits.
Congress established the CWSRF program as a financing mechanism to: (1) replace the
construction grants program and (2) operate in perpetuity delivering subsidies once federal
capitalization ceased. The financial performance of the CWSRF is nothing short of stunning.
Performance measures of special significance include a pace of lending of 93 percent, a federal
return on investment of nearly 2:1, and cumulative available funds that have grown to over
$50 billion in 15 years of operation, all without a single default. Measured by what it was
intended to accomplish as a financing mechanism, the CWSRF has been an unique, unqualified,
and extraordinary success.
20

-------
The benefits of the projects financed by the CWSRF are assessed through state priority
setting systems that rank them based primarily on environmental and public health criteria.
These systems have been in place since 1972 and served essentially the same function for the
construction grants program. We believe that the state priority systems are sound in directing
funding to environmental worthy projects. Since the dominant emphasis is on environmental and
public health benefits, we are confident that the pipeline of projects feeding into the CWSRF is
appropriately results-oriented.
We are focusing our attention, in collaboration with the States, on the actual environmental
results achieved following initiation of operation. A pragmatic and cost-effective way to
approach this challenge is to consider the system of which the CWSRF financed projects are a
part, whether they be pipe or plant or some other eligible construction activity. Projects will be
linked to systems through their NPDES permit. With the permit number other agency databases
can be accessed such as PCS, STORET and water quality standards, using our WATERS
architecture. The opportunities and challenges of this approach and other options will be topics
for discussion at the upcoming ASIWPCA workgroup meetings noted earlier.
I appreciate receiving the report and find it to be informative and supportive of our efforts.
I look forward to the final version. If you have any questions, please contact Sheila Frace at
(202) 564-0749 or George Ames at (202) 564-0661.
cc: Michael Shapiro, Deputy Assistant Administrator, OW
Jeff Peterson, OW
Michael Mason, OW
James Hani on, Director, Office of Wastewater Management
Sheila Frace, Director, Municipal Support Division, OWM
George Ames, Chief, SRF Branch, MSD/OWM
Howard Corcoran, Director, Office of Grants and Debarment
Richard T. Kuhlman, Director, Grants Administration Division
Michael Ryan, Deputy Chief Financial Officer, OCFO
David Ziegele, Director, OCFO/OPAA
Regional Administrators, Regions 1 through 10
Regional Audit Follow-up Coordinators
Nikki Tinsley, Inspector General
Melissa Heist, Assistant Inspector General for Audits
Eileen McMahon, Assistant Inspector General for Congressional and Public Liaison
21

-------
22

-------
Distribution
Appendix E
EPA Headquarters
Assistant Administrator for Water (4101M)
Deputy Assistant Administrator for Water
Director, Office of Wastewater Management
Director, Municipal Support Division, Office of Wastewater Management
Director, Office of Grants and Debarment
Director, Grants Administration Division
Director, Office of Planning, Analysis, and Accountability
Director, Office of Regional Operations (1108A)
Comptroller (2731 A) (2724A)
Agency Followup Official (the CFO) (271 OA)
Agency Audit Followup Coordinator (2724A)
Associate Administrator for Congressional and Intergovernmental Relations (1301 A)
Associate Administrator for Public Affairs (1101 A)
Inspector General (2410)
EPA Regions
Regional Administrators, Regions 1 through 10
Regional Audit Followup Coordinators
23

-------