OIG
OFFICE OF INSPECTOR GENERAL
Ca talyst for Improving the Environment
Memorandum Report
EPA Should Take Further Steps to
Address Funding Shortfalls and
Time Slippages in Permit Compliance
System Modernization Effort
Report No. 2003-M-00014
May 20, 2003
-------
Report Contributors: Dan Cox
Ira Brass
Abbreviations
EPA Environmental Protection Agency
OECA Office of Enforcement and Compliance Assurance
OIG Office of Inspector General
NPDES National Pollutant Discharge Elimination System
PCS Permit Compliance System
-------
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
OFFICE OF
INSPECTOR GENERAL
May 20, 2003
MEMORANDUM
SUBJECT: EPA Should Take Further Steps to Address Funding Shortfalls and
Time Slippages in Permit Compliance System Modernization Effort
Memorandum Report No. 2003-M-000014
FROM: Dan Engelberg /s/
Director for Program Evaluation, Water Issues
TO: John Peter Suarez
Assistant Administrator for Enforcement and Compliance Assurance
This is our final report on the subject review conducted by the Office of Inspector General (OIG)
of the U.S. Environmental Protection Agency (EPA). This memorandum report contains findings
that describe the problems the OIG has identified and corrective actions the OIG recommends.
Due to your failure to sufficiently respond to the draft report, the final report we are sending is
unchanged from the draft. Changes that have occurred since then (specifically, a worsening of
the fiscal situation) are addressed in a separate section at the end of the report. This report
represents the opinion of the OIG, and the findings contained in this report do not necessarily
represent the final EPA position. Final determinations on matters in this report will be made by
EPA managers in accordance with established resolution procedures.
This memorandum report notes significant risks that we have identified regarding your
long-standing program to modernize the Permit Compliance System (PCS). It is essential that
this system, used by the EPA and many States to administer permits for water discharges and
ensure enforcement, be modernized. However, the modernization program is facing a large cost
escalation and a consequent funding shortfall and slippage in time frames. In addition,
consideration is being given to reducing the intended functionality of the system to save money.
As a result, the future viability of PCS may be endangered.
Purpose
The OIG issued an audit report in August 2001, State Enforcement of Clean Water Act
Discharges Can Be More Effective (Report No. 2001-P-00013), that identified weaknesses in the
-------
PCS modernization effort. We made recommendations and suggestions in that report to
improve the modernization effort, and the Office of Enforcement and Compliance Assurance
(OECA) generally agreed with them. Among other things, we suggested that OECA complete a
systems requirements document, cost benefit analysis, system charter, and system management
plan.
Subsequent to our August 2001 report, we have been monitoring the progress of the PCS
modernization project, to help ensure it ultimately meets user needs. The purpose of this
memorandum is to bring to your attention our concerns regarding funding shortfalls, timeframe
slippages, and OECA's failure to complete needed planning documents that might have
improved management decisions for this project.
Background
EPA uses PCS, a national data system, to support the National Pollutant Discharge Elimination
System (NPDES) program. PCS, which is managed by OECA, is a critical information system
for EPA's Office of Water. PCS tracks NPDES permit issuance, permit limits, self monitoring
data, and enforcement and inspection activity for more than 64,000 facilities regulated under the
Clean Water Act. Eighteen States currently use PCS as their major system for managing the
NPDES program and two additional States have expressed an interest in using the modernized
system.
There have been fundamental changes to the NPDES program since the current system was
developed in 1982, including new categories of dischargers for storm water, pretreatment, and
concentrated animal feeding operations. As a result, the Office of Water is planning to bring large
numbers of additional facilities into NPDES, with a consequent need to dramatically expand the
capabilities of PCS. The Office of Water also envisions that PCS will play a central role in
managing its program of watershed-based source controls.
PCS had its last major revision in 1982. It has been identified as an Agency weakness since 1999.
Reasons include its reported unreliability due to missing data and data quality problems.
Moreover, compliance data from hundreds of thousands of smaller dischargers are not captured
by the system.
To improve PCS's ease of use, as well as allow it to catch up to regulatory changes that have
taken place since it was last revised more than 20 years ago, EPA is in the process of modernizing
PCS through a contract with a major consulting company. We noted in our August 2001 report
that the estimated cost for the entire PCS modernization at that time was $12 million to $14
million. The effort started in 1997 and is in the detailed design phase. This phase, which is
currently planned to be completed in May 2003,l is the third of six phases in the modernization
project as shown in the following chart. The detailed design develops the technical
lrThe estimated completion date for detailed design has been extended to September 30, 2003. See
"Developments Since Issuance of Draft Report" on page 9 for more details.
-------
specifications for the system, and includes developing standards, creating the physical data
model, and developing flow charts and screen mock-ups. Detailed design will be followed by the
system development phase when contractors will construct the data model and build and test the
application software. These two phases are the most resource intensive. After several delays,
implementation of the modernized PCS is now planned by the end of fiscal year 2005.2
6. Operations & Maintenance 1 • Concept Definition
(i.e. Administration & Enhancement) (i-e- Reconnaissance)
System
5. System Implementation^ • 2- General Design
(i.e. Install & Acceptance) ^ LlfeCVCle (i.e. Requirements Analysis)
Phases
4. System Development 3- Detailed Design
(i.e. Programming) (i-e Technical Specifications)
Source: NPDES Requirements Document, OECA
Scope and Methodology
Findings from OIG's August 2001 report led to a decision to conduct a followup review of the
PCS modernization effort's project planning and project and contract management. However, in
May 2002, based on a request by OECA, we agreed to monitor the progress of the PCS
modernization process rather than conduct a full review. OECA officials had expressed concerns
about their tight time schedule. We reserved the option to conduct a full review should we
identify material concerns or recognize a situation where the project is in jeopardy.
Our participation consisted of having a representative of our office attend meetings of the PCS
Steering Committee and PCS Executive Council, and provide advice. The PCS Steering
Committee, composed of branch chiefs or equivalent from EPA Headquarters and regions as well
as from States, reviewed the data requirements work groups' recommendations on what data
elements were necessary for effectively managing the NPDES program. The Executive Council,
2The implementation date has now been estimated to be September 30, 2006. See "Developments Since
Issuance of Draft Report" on page 9 for more details.
-------
composed of seven senior EPA managers, reviewed the Steering Committee recommendations
from a broader programmatic perspective. They also compared the requirements for
implementing the recommendations to availability of funding and resources. Based on
information that we gathered through January 2003, we decided to report our concerns in this
memorandum. Since we did not do an audit or evaluation, we did not develop the causes for the
cost increases and time schedule slippage. We complied with all appropriate Government
Auditing Standards. Since we did not conduct an audit or evaluation, we did not comply with
those standards, such as those for planning, compliance with laws and regulations, and
management controls, that apply to those types of products.
Issues of Concern
We identified three issues of concern with respect to the PCS modernization project: (1) cost
estimates recently increased dramatically; (2) the implementation date slipped significantly; and
(3) OECA failed to prepare certain critical project planning documents, and one document
contained inaccurate information.
Funding Shortages
Estimated costs for the PCS modernization project have recently significantly increased. In
November 2002, OECA reported to the PCS Executive Council that the contractor's cost
estimates to finish the detailed design and to design the software had increased by 71 percent.3
The proprietary nature of the current cost estimate means that we are unable to report on the
specific amount of the gap, but we are able to report that it is significant.
Further, in addition to the significant increase in the cost estimate, this estimate still did not
include the cost to conduct certain tasks, including:
• Costs for new functionality approved by the Executive Council, including permit application
and daily monitoring data.
• Costs for the new requirements from the detailed design effort.
• Costs associated with the PCS modernization cost assessment.
• Potential increase in cost estimates for data migration, testing, training, and implementation.
Based on the new cost estimates, which do not include the costs for the above four items, OECA
only has 60 percent of the funding needed to complete the project.
3The cost proposal, submitted in response to OECA's Request for Quote for work to be performed in fiscal
2003, is the contractor's proprietary information, and we are not allowed to report those costs in this memorandum.
The findings reflect internal project cost estimates, but not cost estimates produced in the project's Cost Benefit
Analysis, which are discussed elsewhere in this report.
-------
OECA has several planned approaches for addressing the shortfall, including:
Requesting Additional Funds: OECA plans on funding some of the shortfall from the
Agency's System Modernization Fund. As of February 12, 2003, these funds had not
been distributed for fiscal 2003. The funding shortfall is over half the amount available
from the fund. Members of the council said it was unlikely PCS would receive enough to
fully fund the project. Therefore, the funding level from the fund is uncertain. This is the
last year for the fund.4
Assessing Software Development Costs: OECA plans on conducting a cost assessment
on software development when the detailed design phase is completed in May 2003 to
ensure those increased costs are reasonable. We agree such an assessment should be
conducted, as well as cost estimates for other phases of the project, such as the remainder
of the system development phase and the implementation phase. It is important to
conduct a cost assessment for the remainder of the project because we believe it is likely
that those costs will rise in the future as they have for fiscal 2003. Accurate cost estimates
are needed to signal EPA management of the magnitude of the resources needed as well,
so they can make knowledgeable decisions about funding.
Considering Cost Reductions: Although it has not made any decisions, OECA has
offered several options for reducing costs, including:
• Eliminating some of the new system's functionality.
• Not moving all historical data from the legacy (current) system to the modernized one.
• Using other contractors to perform some of the work.
We have concerns about reducing the functionality of the modernized system and continue to
believe OECA should develop a plan to fully fund the system. Before any functionality,
including transferring all historical data, is eliminated, OECA needs to ensure user needs are still
being met. This includes information from all of
the point sources in a watershed. Information from I „__„ ~~ ~~ ~~~„ ,„ ~~ '
..,,.. . EPA and the states believe that the original user
sources not included in the current system, such as , , ,, , «, t • ,«, n^o
, „ ,. requirements that were the basis of the PCS
storm water, concentrated animal feeding
need reconsideration in light of fundamental
changes to the NPDESprogram."
-OECA's Fiscal 2002 Integrity Act Report
operations, and biosolids, is needed. The major
changes in what is reported in the modernized PCS
are from these areas, and if any of these areas are
not included user needs may not be met. If these
new areas are not included in the modernized
system, information for hundreds of thousands of permittees will not be included. OECA needs
to be very careful before eliminating these areas. Exercising the cost reduction options may
result in a new system that fails to address the basic system requirements that OECA arrived at
4EPA distributed the fiscal 2003 System Modernization Fund subsequent to the issuance of the draft report.
See "Developments Since Issuance of Draft Report" on page 9 for more details.
-------
through consultation with stakeholders. Therefore, before OECA decides to reduce the
functionality of the modernized PCS, it is important that OECA consider the views of the
stakeholders. Also, using other contractors for portions of the work could increase the risk of
further delay in implementation without providing any assurance that it will significantly reduce
costs.
PCS Implementation Date Slips
In addition to the recent cost increases, the PCS implementation date has slipped by about 2
years. OECA's Integrity Act Annual Assurance Letters for fiscal 2000 and 2001 both indicated
the implementation date for the modernized PCS was the fourth quarter of fiscal 2003. OECA's
fiscal 2002 Assurance Letter changed the implementation date to the first quarter of fiscal 2005,
and it is clear based on recent developments that even this date will not be made. OECA officials
most recently said they are planning to implement the system sometime in 2005.5
OECA needs to determine an implementation date that can be met. The goal should include time
to complete cost assessments. OECA should determine different implementation dates for
different funding levels, so the officials who make the funding decisions know the effect their
decisions will have on completing the system.
Required Documents Not Completed and Contain Inaccurate Information
Our third concern is that OECA has directed insufficient attention to conducting accurate and
timely planning and analysis for this project. OECA has not completed certain required planning
documents whose preparation might have improved the management of the program. Further,
another document appears to present significantly inaccurate information.
In our August 2001 audit, we noted that a cost benefit analysis, system charter, and system
management plan for the PCS modernization project had not been prepared. These documents
are used to help managers make informed planning and funding decisions, and are important
elements of quality control for large system acquisition projects, although we cannot say for
certain that completion of these documents would have prevented the funding shortfalls and
delays noted. In its response to our August 2001 audit, OECA agreed to complete these
documents. However, more than 1.5 years after we completed our audit, OECA has prepared
only one of three missing documents - the cost-benefit analysis.
In our prior report, we had identified the following as not being completed:
System Charter and System Management Plan: Although the modernized system was
estimated to cost more than $10 million, the required system charter and system
management plan decision papers had not been prepared or approved. EPA's Information
5 The implementation date has been estimated to be September 30, 2006. See "Developments Since Issuance
of Draft Report" on page 9 for more details.
-------
Resources Management Policy indicates the system charter should have been developed
during project initiation, and it should have identified life cycle cost estimates and
appropriate management levels for approval of decision papers. Further, the decision
paper for the system management plan should have been produced at the conclusion of
the analysis stage and subsequently updated. The system charter becomes a part of the
system management plan. The system management plan includes the items in the system
charter and also includes such items as an acquisition strategy, cost-benefit analysis of the
technical alternatives, and the system's architecture. OECA had indicated it would
establish and approve a system management plan by December 2001, but still has not
done so.
Cost-Benefit Analysis: The detailed design phase for PCS system modernization had
begun without the life cycle cost-benefit analysis required by Office of Management and
Budget Circulars No. A-l 1 and A-130. Such an analysis is necessary to identify the most
cost-effective solution for the new system and expected benefits. OECA had indicated it
would complete the analysis by September 2001. OECA completed the cost-benefit
analysis almost a year later, in August 2002. Although we did not conduct a thorough
review of the analysis, our brief review suggests that this important analysis greatly
understated the costs to finish the project. [Again, the proprietary nature of the current
cost estimate means that we are unable to report on the specific amount of the increase
in cost.] Specifically, we noted:
• In September 2002, 1 month after the analysis was completed, OECA developed an
internal cost estimate 171 percent higher than had been published in the analysis.
• In November 2002, 2 months later, OECA prepared another internal cost estimate that
projected costs to increase even more. This cost estimate, 3 months after the August
estimate, was 255percent higher than that contained in the cost-benefit analysis.
Performing the analysis utilizing the November 2002 cost data as an input would have
resulted in a very different cost-benefit ratio. As a result, inaccurate information was
provided to Agency managers and made available to the public, which may have delayed
the enhanced attention and management assistance needed for this project.
Conclusion
Without a modernized PCS, EPA's Office of
Water cannot effectively manage its Clean Water
NPDES program. Having a modernized system
is vital for EPA to effectively manage NPDES
permitting and enforcement under current
requirements. The current system is incomplete,
obsolete, and difficult to use. The glaring
weaknesses in the current PCS system have
created a presumption in EPA that it will
"We find ourselves in the Age of Information with a
dearth of essential, scientifically defensible data and
information to manage our programs. It is imperative
that we close these information gaps as quickly as
possible: they may lead to market and regulatory
failures, thwart our ability to document progress, and
limit our ability to effectively target our scarce
resources."
-Office of Water's Fiscal 2004 Priorities for Regions
-------
be modernized. We agree with EPA's view of the importance of this project, and believe delaying
the project's rollout or reducing its functionality will hamper EPA's ability to achieve its goal of
managing pollution sources on a watershed basis. The growth, variety, and complexity of the
regulated community has greatly outstripped the system's capabilities.
However, costs are dramatically escalating, and timeframes repeatedly pushed back, in part due to
the failure to adequately plan, prepare, and manage the work. The critical role of the modernized
PCS system does not make project management unimportant. On the contrary, management risks
may be greater when a project is perceived as being vital. For this reason, it is imperative that EPA
immediately conduct necessary analyses and develop realistic estimates of funding and schedules
in order to place this project on a secure footing.
Recommendations
We recommend that the Assistant Administrator for Enforcement and Compliance Assurance:
1. Develop a realistic cost estimate and cost-benefit analysis that includes all of the projected
costs for PCS modernization and develop a plan for fully funding PCS within 30 days of
the date of this report.
2. Consult with the PCS Steering Committee before eliminating any functionality of
modernized PCS and conduct a cost-benefit analysis to determine what is lost, what is still
gained through the modernized system, and whether the modernization effort still meets
the needs of the users and is worthwhile to continue.
3. Develop realistic implementation dates for different funding levels within 30 days of the
date of this report.
4. Complete the system management plan within 30 days of the date of this report.
Agency Comments
We received an interim response to our draft report from OECA. In its response, OECA did not
respond directly to our recommendations, but indicated it is beginning to take steps that it believes
will begin to address our concerns. OECA stated that it did not have time to fully respond to our
draft report, since it is focusing its efforts on completing the draft PCS Modernization Detail
Design Document by the end of May 2003. After that, OECA said it will fully respond to our
report. OECA stated it has started addressing our recommendations, and provided three examples
of how it has started to address the funding shortfall.
OECA's full comments are in Appendix A.
-------
Developments Since Issuance of Draft Report
We are adding this section of the report because actions have taken place since we issued the draft
report. OECA did not address them in its interim response and we are not waiting for its final
response to issue the report.
EPA has distributed the fiscal 2003 System Modernization Fund, and OECA received significantly
less from the fund than anticipated. There was also a significant decrease in fiscal 2003 OECA
funding. OECA had hoped that the distribution from the fund would have significantly decreased
the 40 percent funding shortfall discussed in the draft report. Instead, it only decreased the
shortfall to 33 percent.
A large portion of the planned funding is in EPA's fiscal 2004 budget that was submitted to
Congress but has not been approved. If it is not approved, OECA's funding shortfall increases to
62 percent from the 33 percent shortfall discussed in the previous paragraph.
OECA is developing plans to request funds from the fiscal 2005 budget to fund much of the
shortfall. If the funding is approved, the new date for implementing the modernized system will
slip one additional year to September 30, 2006. This is 3 years after the original implementation
date.
Also, the detailed design completion date has slipped from May 31, 2003, to September 30, 2003,
a full 2 years after what OECA reported in its fiscal 2000 Integrity Act Annual Assurance Letter.
Required Actions
In accordance with EPA Manual 2750, you are required to provide a written response to this
report within 90 calendar days of the date of this report. You should include a corrective actions
plan for agreed upon actions, including milestone dates. We have no objections to the further
release of this report to the public. For your convenience, this report will be available at
http://www.epa.gov/oig/eroom.htm.
If you or your staff have any questions, please call me at (202) 566-0830, or Dan Cox, Project
Manager, at (916) 498-6592.
-------
10
-------
Appendix A
Agency's Response
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
May 8, 2003
OFFICE OF
ENFORCEMENT AND
COMPLIANCE ASSURANCE
MEMORANDUM
SUBJECT: EPA Should Take Further Steps to Address Funding Shortfalls and Time Slippages
in Permit Compliance System Modernization Effort
DRAFT: Memorandum Report No. 2003-M-OOOOXX
FROM: Phyllis P. Harris /s/
Principal Deputy Assistant Administrator
TO: Dan Engelberg
Director for Program Evaluation, Water Issues
Office of the Inspector General
We are working to respond to your April 2, 2003 draft memorandum report titled, "EPA
Should Take Further Steps to Address Funding Shortfalls and Time Slippages in Permit
Compliance System Modernization Effort." Your office identified three issues of concern
regarding PCS modernization and provided four recommendations for addressing those concerns.
We intend to fully respond to your draft report, but at this time we are focusing our efforts on
completing the draft PCS Modernization Detail Design Document. As you know, the critical path
for the PCS modernization effort is to complete the draft Design Document by the end of May
2003. If we took the time now to fully respond to your draft report, we would endanger our ability
to deliver the Detail Design.
We have started to address your recommendations. For example, we are developing revised
scenarios for implementing a modernized PCS within certain time frames based on different
funding levels. We are also reviewing whether the modernized PCS could be phased in over
11
-------
time, perhaps with two versions, and how this could be done with regards to functionality,
schedules and budget. In addition, we are developing plans for obtaining the necessary funding,
including developing OECA's FY2005 Integrated Compliance Information System (ICIS) Capital
Planning and Investment Control (CPIC) proposal (includes PCS modernization).
We anticipate providing a full response to your concerns and recommendations by mid-June
2003. This will allow staff who are now focused on completing the Design Document to finish
that work and then focus on responding fully to your report. Please call me if you have
any questions or contact David Hindin, Acting Director, Enforcement Planning, Targeting and
Data Division, at (202) 564-1300.
cc: Michael Stahl, OECA/OC
Jim Hanlon, OW/OWM
Mark Luttner, OEI/OIC
David Hindin, OECA/OC
Greg Marion, OECA/ARMSS
12
-------
Appendix B
Distribution
Headquarters Offices
Assistant Administrator for Enforcement and Compliance Assurance
Assistant Administrator for Water
Comptroller (2731 A)
Agency Followup Official (2710A)
Agency Audit Followup Coordinator (2724A)
Office of Enforcement and Compliance Assurance Audit Liaison (2201 A)
Associate Administrator for Congressional and Intergovernmental Relations (1301A)
Associate Administrator for Communications, Education, and Media Relations (1101 A)
Office of Inspector General
Inspector General
13
-------
------- |