\saA OIG
% PPO^ OFFICE OF INSPECTOR GENERAL
Catalyst for Improving the Environment
Audit Report
EPA Could Increase Savings and
Improve Quality Through Greater Use of
Performance-Based Service Contracts
Report No. 2003-P-00008
March 31, 2003
-------
Report Contributors: Herbert Maletz
Denton Stafford
Charles Watts
Brian Lick
Abbreviations
EPA Environmental Protection Agency
FAR Federal Acquisition Regulation
FPDS Federal Procurement Data System
GAO General Accounting Office
ICMS Integrated Contracts Management System
OIG Office of Inspector General
OMB Office of Management and Budget
PBSC Performance-Based Service Contracting
START Superfund Technical Assessment and Response Team
-------
v. UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
'. WASHINGTON, DC 20460
'"(
OFFICE OF
INSPECTOR GENERAL
March 31,2003
MEMORANDUM
SUBJECT: Audit Report:
EPA Could Increase Savings and Improve Quality
Through Greater Use of Performance-Based Service Contracts
Report No. 2003-P-00008
FROM: Robert Mitchell
Director for Contract Audits
TO: Morris X. Winn
Assistant Administrator
Office of Administration and Resources Management
This is our final report on the subject audit conducted by the Office of Inspector General (OIG)
of the U.S. Environmental Protection Agency (EPA). This audit report contains findings that
describe the problems the OIG has identified and corrective actions the OIG recommends. This
audit report represents the opinion of the OIG and the findings contained in this report do not
necessarily represent the final EPA position. Final determinations on matters in this audit report
will be made by EPA managers in accordance with established audit resolution procedures.
Action Required
In accordance with EPA Order 2750, you are required to provide a written response to the
findings and recommendations presented in this draft report within 90 calendar days of the date
of this report. You should include a corrective actions plan for agreed-upon actions, including
milestone dates. We have no objection to the further release of this report to the public. For your
convenience, this report will be available at http://www.epa.gov/oig/eroom.htm.
If you or your staff have any questions regarding this report, please contact me at (202) 566-0891
or Herbert Maletz at (212) 637-3058.
-------
-------
Executive Summary
Purpose
Performance-Based Service Contracting (PBSC)is an acquisition methodology in
which the Government pays for results rather than effort or process. Office of
Management and Budget (OMB) studies have shown that PBSC saves money and
improves contractor performance. While PBSC has existed for many years, it has
not been widely adopted throughout the Federal Government.
We conducted this audit to determine whether the Environmental Protection
Agency (EPA) effectively implemented PBSC to the maximum extent possible.
Our specific objectives were to determine whether EPA was:
" Awarding performance-based service contracts whenever possible and
designing the contracts to take full advantage of the benefits.
" Reporting accurate PBSC information to OMB.
Results of Review
EPA has made limited use of PBSC. Generally, EPA has only used PBSC to
obtain commercially available services, such as janitorial and landscaping work.
We found that EPA awarded contracts that were not performance-based for
services previously awarded as performance-based. Also, many contracts EPA
identified as performance-based were not designed to take advantage of PBSC s
benefits. These contracts were too prescriptive in how the work was to be carried
out or did not provide meaningful incentives or disincentives. Based on
discussions with Agency officials, this occurred because EPA did not adequately
plan or hold officials accountable for increasing PBSC use, and there was a
general reluctance to shift the responsibility for outcomes to contractors. As a
result, EPA missed opportunities to achieve greater cost savings and improve
contractor performance. Of the $599 million in PBSC-eligible obligations for the
9-month period reviewed, we determined that EPA could have saved as much as
$72.5 million by better applying PBSC.
Also, OMB s Federal Procurement Data System (FPDS) showed nearly 50 percent
less PBSC obligations than EPA reported in its own Integrated Contracts
Management System (ICMS). This primarily occurred because FPDS did not
show performance-based task orders issued under non-performance-based
contracts as PBSC; EPA made the necessary corrections in ICMS but had not
submitted them to FPDS. In addition, EPA officials said ICMS lacked the
capability to provide all data needed in FPDS. As a result, EPA s contracting
-------
actions were inaccurately portrayed in the national database, which serves as a
basis for reporting to Congress, the President, and the general public; and for
performing analysis.
Recommendations
We recommend that the Assistant Administrator for Administration and
Resources Management take certain actions to improve accountability and
procedures in relation to PBSC. We also recommend that the Assistant
Administrator take actions so that the accuracy of ICMS is reviewed and that
ICMS and FPDS data is periodically compared.
Agency Comments and OIG Evaluation
On February 14, 2003, we issued a draft report to the Office of Administration
and Resources Management, and received a response on March 19, 2003
(see Appendix B). The Office generally agreed with the findings and
recommendations and included planned corrective actions. The Office
acknowledged significant problems with contract tracking systems, and noted a
new system it has proposed will correct the problems. In its response, the Office
also provided additional information, and we clarified our final report as needed.
-------
Table of Contents
Executive Summary
Chapters
1 Introduction 1
2 EPA Needs to Increase Use of PBSC 3
3 EPA Needs to Improve PBSC Reporting 13
Appendices
A Details on Scope and Methodology 17
B Agency Response to the Draft Report 19
C Report Distribution 25
-------
-------
Chapter 1
Introduction
Purpose
Performance-Based Service Contracting (PBSC) is an acquisition methodology in
which the Government pays for results rather than effort or process. Office of
Management and Budget (OMB) studies have shown that PBSC saves money and
improves contractor performance. While PBSC has existed for many years, it has
not been widely adopted throughout the Federal Government.
We conducted this audit to determine whether the Environmental Protection
Agency (EPA) effectively implemented PBSC to the maximum extent possible.
Our specific objectives were to determine whether EPA was:
" Awarding performance-based service contracts whenever possible and
designing the contracts to take full advantage of the benefits.
" Reporting accurate PBSC information to OMB.
Background
PBSC allows contractors to propose the most innovative, effective, efficient, and
economical approach their business experience and ability will allow. Such
contracts can help an agency avoid cost overruns, schedule delays, and failure to
achieve specified results. EPA participated in OMB s PBSC studies as well as its
own pilot projects and has experienced these benefits.
OMB s Office of Federal Procurement Policy has encouraged greater use of PBSC
for over 10 years. In April 1991, this Office issued Policy Letter 91-2, Service
Contracting. This Policy Letter defined PBSC, required the use of PBSC where
practicable, and required agencies to better match their acquisition and contract
administration strategies to the specific requirements.
Policy Letter 91-2 was incorporated into the Federal Acquisition Regulation
(FAR) in 1997. Subpart 37.1 of the FAR, Service Contracts-General, states when
acquiring services, agencies must use performance-based contracting methods to
the maximum extent practicable, except for architect-engineer services,
construction, utility services, and services incidental to supply purchases.
-------
FAR Subpart 37.6, Performance-Based Contracting, first issued in 1997,
prescribes policies and procedures for PBSC. The FAR notes performance-based
contracts:
" Describe the requirements in terms of results required rather than the methods
of performing the work;
" Use measurable performance standards (i.e., terms of quality, timeliness,
quantity, etc.) and quality assurance surveillance plans;
" Specify procedures for reductions of fee or for reductions to the price of a
fixed-price contract when services are not performed or do not meet contract
requirements; and
" Include performance incentives where appropriate.
OMB s Office of Federal Procurement Policy issued Information on Best
Practices for Performance-Based Service Contracting in October 1998. This
guide notes studies have shown that PBSC has the potential to improve customer
satisfaction by more than 18 percent and achieve average contract cost savings of
15 percent. However, the Office noted that application of only selected aspects of
the PBSC methodology can limit benefits.
Agencies must report PBSC information to the Federal Procurement Data System,
which is the central repository of statistical information on Federal contracting.
The system contains detailed information on contract actions over $25,000, and
summary data on procurements less than $25,000.
Scope and Methodology
From October 2001 through June 2002, EPA acquired approximately
$599 million in services that were PBSC-eligible. This information was obtained
from the November 7, 2002, Integrated Contracts Management System (ICMS).
We performed this audit in accordance with Government Auditing Standards,
issued by the Comptroller General of the United States. We judgmentally selected
a sample of 30 contracts and task orders for review. We assessed EPA s policies
and procedures to ensure performance-based contracts were awarded when
possible, determined whether contracts statements of work were designed with
performance-based attributes, and determined whether PBSC information was
reported accurately. During the audit, we interviewed numerous personnel. We
performed this audit from December 2001 through November 2002. As part of
our review, we took into account a prior General Accounting Office (GAO) report
(GAO-02-1049, Guidance Needed for Using Performance-Based Service
Contracting, September 2002). Details on our scope and methodology are in
Appendix A.
-------
Chapter 2
EPA Needs to Increase Use of PBSC
EPA has made limited use of PBSC. Generally, EPA has used PBSC to obtain
commercially available services, such as janitorial and landscaping work. We
found that EPA awarded non-performance-based contracts for services previously
awarded as performance-based. Also, many contracts EPA identified as
performance-based were not designed to take advantage of PBSC s benefits.
These contracts were too prescriptive in how the work was to be carried out or did
not provide meaningful incentives or disincentives. Based on discussions with
Agency officials, these conditions occurred because EPA did not adequately plan
or hold officials accountable for increasing PBSC use, and there was a general
reluctance to shift the responsibility for outcomes to contractors. As a result, EPA
missed opportunities to achieve greater cost savings and improve contractor
performance. Of the $599 million in PBSC-eligible obligations for the 9-month
period ended June 30, 2002, we determined that EPA could have saved as much
as $72.5 million by better applying PBSC.
OMB Has Encouraged Greater Use of PBSC Since 1991
As noted in Chapter 1, OMB has required the use of PBSC where practicable
since 1991. In 1997, this requirement was incorporated into the FAR,
Subpart 37.6, Performance-Based Contracting. PBSC is intended to ensure that
required performance quality levels are achieved and that total payment is related
to the degree that services performed meet contract standards. Performance-based
contracts describe the desired end result (outcomes) expected of the contractor,
but do not direct how the work is to be performed.
In April 2002, FAR Subpart 7.105, Contents of Written Acquisition Plans,
required that acquisition planners develop written acquisition plans for service
contracts or task orders that describe strategies for implementing performance-
based contracting methods, or provide rationale for not using those methods.
EPA s Contracts Management Manual, Chapter 1 (February 1999) notes EPA s
Senior Resource Officials are the primary acquisition planners.
EPA Has Made Limited Use of PBSC
Generally, EPA has limited PBSC to services performed widely in the commercial
sector: custodial services, building maintenance, or landscaping. As noted in a
GAO report, this was the case with many agencies. These services easily lend
themselves to PBSC because measuring and specifying outcomes are relatively
straightforward. Additionally, these generally non-complex services do not
present substantial risk to the Government.
-------
OMB has set Government-wide goals for PBSC. Beginning in fiscal 2001,
10 percent of all eligible service contracting obligations were to be performance-
based. The PBSC goals incrementally increase 10 percent annually until 2005.
The goal for fiscal 2002 was 20 percent.
For fiscal 2001, EPA reported approximately 11 percent of its eligible contract
obligations were performance-based, and thus met its goal of 10 percent.
However, for fiscal 2002, EPA only reported that 16.6 percent of contract
obligations were performance-based, short of OMB s goal of 20 percent. EPA
needs to increase its effort to meet OMB s PBSC goals.
EPA Could Award More Performance-Based Contracts
Between 1998 and 2001, EPA awarded many contracts without employing PBSC
attributes, for services similar to previously awarded EPA performance-based
contracts. We reviewed 10 such contracts (with a maximum potential value of
$16.3 million), as shown in Table 2-1:
Table 2-1: Contracts Not Performance-Based for Which Similar Services Were
Previously Awarded as Performance-Based
Contract Maximum
Number Service Potential Value
68-D-01-051
68-W-00-128
68-W-00-125
68-W-99-078
68-D-99-026
68-D-99-014
68-W-99-012
68-D-98-166
68-D-98-092
68-D-98-029
Total
Cafeteria Services and Operation of EPA
Employee Store
Voice Mail System Maintenance
Technical Supportfor Compliance Assurance
Facility Operatbn and Maintenance Service for trie
Manchester EPA Laboratory
Landscape/Grounds Maintenance Support
National Performance Audit Program
(Air Monitoring)
Perimeter Guard Services
Security Guard Services
Peer Review, Workshops, Training Sessions, and
Conference Support
Maintenance Services for the Natbnal Airand
Radiation Laboratory
$264,000
677,000
2,813,000
916,000
162,000
7,127,000
3,218,000
505,000
292,000
284,000
$16,258,000
Most of these contracts were for services widely performed in the commercial
sector, and did not present substantial risk to the Agency. Measuring and
specifying outcomes for these types of services were relatively straightforward.
Based on the types of services acquired and the fact that EPA had previously
awarded a performance-based contract of a similar nature, we believe these
contracts could have been performance-based.
-------
EPA Office of Acquisition Management officials said the above contracts were
not performance-based for a variety of reasons, including urgency of need,
inability to define performance objectives, and inexperience in using PBSC
methods (some earlier contracts). The officials also stated the Voice Mail System
Maintenance contract was not fully performance-based primarily because staff did
not have enough time to convert the statement of work.
In addition to the 10 non-performance-based contract files reviewed, we also
selectively reviewed ICMS data as of June 30, 2002, and found instances where
EPA awarded non-performance-based contracts even though contracts for similar
services were previously performance-based:
" ICMS reported four non-performance-based facilities operation support
services contracts, even though at least seven other facilities operation support
services contracts within the past 5 years were performance-based.
" Two landscaping contracts were not identified as performance-based, even
though in the last 2 years EPA awarded at least two other landscaping
contracts identified as performance-based.
" EPA awarded a janitorial/custodial contract not identified as performance
based, even though within the last 5 years EPA awarded at least nine other
performance-based janitorial/custodial contracts.
" EPA awarded a non-performance-based mail service contract, even though
within the last 5 years EPA has awarded at least five other performance-based
mail services contracts.
Performance-Based Contracts Not Always Designed to Take
Advantage of Benefits
Performance-based contracts were not always designed to take advantage of
PBSC benefits. We reviewed 20 individual performance-based contracts and task
orders ($77.8 million maximum potential value), which generally contained at
least one of the following performance-based attributes required by FAR,
Subpart 37.6:
" Performance requirements described in terms of results required.
" Measurable performance standards and quality assurance surveillance plans.
" Financial disincentives.
" Performance incentives.
Eight of the contract actions appeared to generally be designed to take advantage
of PBSC benefits, even though they may not have contained all four of the above
-------
PBSC attributes. However, in our opinion, the other 12 contracts were not
designed to take advantage of PBSC s benefits. These contracts PBSC attributes
were either prescriptive in how the work was to be carried out or did not provide
meaningful incentives to increase contractor efficiency or penalize contractors for
poor performance.
Performance-Based Contracts Designed to Take Advantage of PBSC
Benefits
The following eight performance-based contracts appeared to generally be
designed to take advantage of PBSC benefits (see Table 2-2). These contracts
exhibited many of the basic PBSC attributes.
Table 2-2:
Performance-Based Contracts Designed to Take Advantage of PBSC Benefits
Contract
Number
68-C-01-025
68-D-01-049
68-S3-00-02
Work Assmnt. #
SE3-02-03-010
68-W-00-075
68-W-00-095
68-W-01-018
Task Order # 005
68-W-01-059
68-W-02-020
Total
Maximum
Service Potential Value
Study to Determine Concentrations of
Pollutants in Waste Water
Janitorial Services for Facilities
START (Superfund Technical Assessment and
Response Team) Removal Site Evaluation
Contract Laboratory Program-Analytical
Support
Energy Star Hotline and Distribution Center
ESAT (Environmental Services Assistance
Team) Performance Evaluation and Field
Support Services
Warehousing and Storage Services
Property Management Services
$48,500
732,000
39,000
2,245,000
2,982,000
8,148,000
13,557,000
2,117,000
$29,868,500
The START Removal Site Evaluation contract action illustrates how PBSC can
provide benefits. EPA only specified the desired end result, not the process to be
used. EPA obtained two competing proposals and found both to be technically
sound. However, while one firm proposed using standard sampling technology,
the other firm proposed using a new technology that had proven successful at
another site. The proposals differed significantly in estimated costs: the estimate
for the standard sampling technology was $154,000 while the new technology
estimate was $39,000, or $115,000 less. EPA selected the lower cost proposal.
By not being prescriptive, EPA opened the door to more innovative approaches
that cost less and possibly improved service.
-------
While the eight contract actions in Table 2-2 represent EPA s best PBSC efforts,
it is not our intent to suggest that the attributes could not be improved.
Contracts That Were Prescriptive
Seven contracts/actions, although identified as performance-based, were
prescriptive in how the work was to be performed, and frequently required
direction from EPA project officers (see Table 2-3). Most of these contracts were
for traditional, relatively low-risk services.
Table 2-3: Contracts That Were Prescriptive
Contract Maximum
Number Service Potential Value
68-C-00-137
68-C-00-189
68-D-98-169
68-D-00-200
68-S3-99-04
Task Order # 12
68-W-99-082
68-W-01-002
Total
Study to Estimate Selected Chemical Levels in
Fish Tissue
Security Guard Services
Security Guard Services
Facility Maintenance and Equipment Operation
of Lab
ERRS (Emergency Rapid Response Service)
Contamination Removal
Administrative Support Services
Information Management Support Services
$13,800,000
3,000,000
774,000
4,600,000
1,900,000
2,300,000
343,000
$26,717,000
On the surface, these contracts appeared to contain the PBSC attributes.
However, a closer review indicated prescriptive statements of work that might
have impeded the contractor s innovation and freedom to decide how best to
perform the work. For example, one guard services contract (68-C-00-189)
prescribed the number of guards desired, specific posts to be manned, specific
sizes for uniform patches and vehicle lettering, and specific age of all guard
vehicles. These examples illustrate EPA s reluctance to allow contractors the
flexibility to use innovative approaches or shift responsibility for outcomes.
Incentives and Disincentives Not Meaningful
Five contracts/actions, which EPA identified as performance-based, did not have
meaningful incentives and/or disincentives. Although two provided negative
incentives, the proposed penalties were low dollar amounts. The other three did
not include any incentives (see Table 2-4).
-------
Table 2-4: Incentives/Lack of Incentives Not Meaningful
Contract Maximum
Number Service Potential Value
68-C-99-252
68-C-99-254
68-W-00-104
68-W7-0055
Task Order #408
68-W7-0055
Task Order #422
Total
Environmental Impact Analysis
Administrative Support Services
Security Support Services
Library Systems Support
Operations of National Computer Center
$1,996,000
2,573,000
11,413,000
101,000
5,106,000
$21,189,000
For example, while the administrative support services contract (68-C-99-254)
exhibited PBSC attributes, its negative incentives were not meaningful. The tasks
were organized in such a way that minuscule penalties could be assessed.
Specifically, the penalty assessed against poor performance for inventorying the
stockroom was 0.4 percent of $4,004.33 per month, or $16.02.
Superfund PBSC
While most of EPA s current performance-based contracts are for less complex
and lower risk services that are widely available in the commercial sector, its
largest contracts have been more complex, higher risk, cost reimbursable, level of
effort types, especially for the Superfund Program. These contracts place more of
the risk for cost and performance on the Government; contractors are only
required to give their best effort, and all reasonable and allowable costs are
reimbursed.
The Office of Acquisition Management and Office of Emergency and Remedial
Response developed PBSC model statements of work templates for various
Emergency Rapid Response Service, START, and Response Action Contract
assignments and tasks that take place on a regular basis. According to Office of
Emergency and Remedial Response officials, the overall structure of these
multi-year time and materials contracts was not developed using the PBSC
methodology, and the use of these templates was optional. These templates were
developed in 2002 as a temporary measure to increase PBSC use under the current
contracts, and were not widely used at the time of our review. Further, these
templates by design did not significantly change current contract management
practices of relying upon close EPA contractor oversight. Therefore, until the
basic contract incorporates all PBSC features and is managed accordingly, EPA
may not maximize PBSC benefits.
-------
More PBSC Planning and Accountability Needed
To promote the use of PBSC, EPA s Office of Acquisition Management relied on
a marketing strategy involving training and assistance to program offices.
However, EPA did not adequately plan or hold responsible officials accountable
for increasing PBSC s use. We believe this is primarily due to Agency officials
reluctance to shift the responsibility for outcomes to contractors.
Current PBSC Planning Not Specific Enough
To meet the PBSC goals, the Office of Acquisition Management developed a
short-term plan for fiscal 2002, which focused on:
" Sharing responsibility for meeting the goal with program office customers;
" Implementing additional, more focused PBSC training;
" Providing improved PBSC outreach, direction, and tools to make PBSC more
user friendly to EPA offices;
" Converting additional program requirements to PBSC; and, ultimately,
" Sharing lessons learned and best practices throughout the Agency.
This plan generally emphasized greater PBSC use throughout EPA, but there was
no high-level accountability to ensure the plan s implementation. The plan did
not outline a detailed strategy to identify additional program requirements that
could be converted to PBSC, although such a strategy is planned for 2003.
Furthermore, since EPA contracts generally last 3 years or longer, EPA s plan
should also include a long-term PBSC strategy to address contracts that will not
need to be replaced for some time.
EPA s Acquisition Forecast Database could help identify planned performance-
based contracts. This web site provides the business community with information
about upcoming EPA procurements. As of September 4, 2002, of the 204 planned
contracts listed, 38 (19 percent) were identified as performance-based. Our
review of the list showed that some traditional services, such as library operations,
were not identified as performance-based.
Lack of Accountability For PBSC Within EPA
EPA s Office of Acquisition Management has attempted to market and promote
PBSC, but has experienced difficulties with obtaining firm program office
commitments on future performance-based acquisitions. The Office s staff said
this is due to the general reluctance of program officials to change their current
culture of managing large, cost-reimbursable, level-of-effort-type contracts.
Program officials echoed this sentiment, expressing concerns about major changes
and the belief that they would be losing control of their business. While the
Office of Acquisition Management is primarily responsible for all EPA
-------
contracting activity, the Agency has not held program officials accountable to
ensure the adoption and implementation of PBSC agency-wide. For example,
another agency (the U.S. State Department) issued a directive that all new service
contracts must be performance-based, with any deviations justified in writing and
approved. EPA may need such a commitment from their Senior Resource
Officials, the primary acquisition planners.
EPA Pilot Project Exhibited Benefits
EPA was part of OMB s PBSC pilot project from 1994 to 1998 to encourage
Government-wide acceptance of PBSC and to evaluate its effects. EPA
participated with two contracts one for operation of a telephone hotline and the
other for training. The contract price was reported to decrease by 31 percent for
the hotline and 6 percent for training. In Fiscal 1997, with assistance from OMB,
EPA conducted a PBSC study within the Superfund program. Nine projects were
involved in this study, and most appeared to save money. For example, at the
Oronogo-Duenweg Mining Site, PBSC was responsible for lowering the project
costs by 30-35 percent while improving the quality of work.
Significant Savings Could Be Achieved
During our review, we found that EPA missed opportunities to achieve greater
cost savings and higher quality service. For the 9-month period reviewed
(October 1, 2001, through June 30, 2002), EPA reported in ICMS (as of
November 7, 2002) that it obligated a total of $599 million for eligible
performance-based services, with $116 million labeled performance-based. OMB
studies have shown that PBSC has the potential to improve customer satisfaction
by more than 18 percent, and achieve average contract cost savings of 15 percent.
Applying this potential savings to EPA s remaining obligations eligible to be
performance-based ($483 million) indicates that EPA could have saved as much
as $72.5 million during this 9-month period while improving the quality of
services obtained.
Recommendations
We recommend that the Assistant Administrator for Administration and
Resources Management, in coordination with the Senior Resource Officials for
each Headquarters and regional office and other field locations, promote
maximum use of PBSC by:
2-1. Developing and implementing a PBSC plan that includes short-term and
long-term strategies for maximizing the use of performance based
contracting. The plan should establish goals for PBSC use in program
offices and monitor progress against those goals.
10
-------
2-2. Developing and implementing procedures to improve the use of PBSC,
such as deploying experienced PBSC contracting personnel to assist
program offices in developing performance-based statements of work, and
ensuring that the statements of work contain the required elements.
2-3. Identifying forums, such as the Innovation Action Council, to help further
promote PBSC, and to discuss PBSC best practices and issues on a regular
basis.
2-4. Developing and implementing a procurement policy addressing the FAR
Subpart 7.105 requirement to implement performance-based contracting
methods, or provide a rationale for not using those methods.
Agency Comments and OIG Evaluation
The Office of Administration and Resources Management agreed with the
findings and recommendations and included planned corrective actions. The
Office also provided additional information, and we clarified the report as needed.
The full response is included in this report as Appendix B.
11
-------
12
-------
Chapter 3
EPA Needs to Improve PBSC Reporting
OMB s Federal Procurement Data System (FPDS) showed nearly 50 percent less
PBSC obligations than EPA reported in its own ICMS. This primarily occurred
because FPDS did not show performance-based task orders issued under non-
performance-based contracts as being performance-based. EPA made the
necessary corrections in ICMS but had not submitted them to FPDS. In addition,
EPA officials said ICMS lacked the capability to provide all data needed in FPDS.
As a result, EPA s contracting actions were inaccurately portrayed in the national
database, which serves as a basis for reporting to Congress, the President, and the
general public, and for performing analysis.
Requirements
Public Law 93-400, Office of Federal Procurement Policy Act, requires the Office
of Federal Procurement Policy to establish a system for collecting, evaluating, and
disseminating Federal contract information. FPDS is that central repository.
FAR Subpart 4.6, Contract Reporting, requires each executive agency to establish
and maintain for a period of 5 years a computer file, by fiscal year, containing
unclassified records of all procurements exceeding $25,000. Agencies must
transmit this information to FPDS.
The FPDS Reporting Manual outlines PBSC information reporting requirements,
and states that PBSC applies only to specific service codes. Any procurement
action coded as a product or as any of the following services is not eligible to be
categorized as PBSC in FPDS:
" Architect-engineer services.
" Construction.
" Utility services.
" Services that are incidental to supply purchases.
FAR Subpart 37.102 states agencies are not required to use performance-based
contracting methods for these services. For PBSC reporting purposes, these
services are considered not PBSC eligible.
ICMS is EPA s automated acquisition system and contains information on EPA s
contracting activities. EPA converts ICMS information to a format compatible
with FPDS, and this information is then sent to FPDS quarterly. To report accurate
PBSC information, the Agency must determine whether the procurement action
13
-------
meets PBSC eligibility requirements, contains required attributes, and uses an
appropriate service code.
EPA Under-Reported PBSC Activity
Our comparison of ICMS and FPDS summary level data for October 1, 2001, to
June 30, 2002, disclosed that almost 50 percent of EPA s PBSC information was
not shown as such in FPDS (see Table 3-1). EPA s total obligations in the two
systems, as well as its PBSC-eligible obligations, were within 10 percent of each
other.
Table 3-1: Comparison of ICMS and FPDS Data ($ in millions)
EPA Obligations
Total Obligations
Not PBSC Eligible
PBSC Eligible
PBSC
ICMS
$649
50
599
$116
FPDS
$594
45
549
$61
Percent
Under-Reported
8.5%
10.0%
8.0%
47.5%
The PBSC category are obligations for procurement actions that EPA has
determined met PBSC requirements.
Performance-Based Task Orders Issued Under Non-Performance-
Based Contracts Not Shown as PBSC in FPDS
FPDS reported $55 million less PBSC obligations than ICMS. ICMS shows
performance-based task orders issued under non-performance-based contracts, but
these actions were not shown in FPDS as PBSC. ICMS reported approximately
$43 million in performance-based task order obligations issued under non-
performance-based contracts that were not shown as PBSC in FPDS. For example,
ICMS reported that a national telecommunication and computer services contract
(68-W7-0055) contained approximately $29 million of performance-based task
orders, but that amount was not shown as such in FPDS.
EPA Office of Acquisition Management officials acknowledged ICMS and FPDS
inconsistencies in reporting PBSC information, and stated that ICMS information
as of June 30, 2002, which was sent to FPDS, contained these errors. System users
subsequently corrected the ICMS information, but EPA had not submitted these
corrections to FPDS at the time of our review. These EPA officials said system
users data entry is normally not reviewed for accuracy. The Office of Acquisition
Management s fiscal 2002 Assurance Letter under the Federal Manager s Financial
Integrity Act indicated that data quality is a major concern in EPA, and they are
currently investigating these problems and are committed to their resolution.
14
-------
Other PBSC Reporting Inconsistencies
Another FPDS database reporting inconsistency concerns work assignments issued
under EPA contracts. EPA designed work assignments to control work flow but,
unlike task orders, work assignments are not separate funding instruments from the
basic contract. Therefore, FPDS does not recognize work assignments.
Consequently, EPA performance-based obligations organized under work
assignments in ICMS were not reported in FPDS as a performance-based
obligation.
Further, work assignments issued under architectural and engineering services
contracts are not eligible in FPDS to be performance-based due to the service code
of the basic contract. The architectural and engineering Response Action Contract
for EPA Region 5 included two performance-based work assignments with a
combined value of $1.3 million. While these obligations were shown in ICMS as
performance-based, they were not reported in FPDS as being performance-based.
The Office of Acquisition Management s fiscal 2002 Assurance Letter notes ICMS
cannot be used to accurately identify and report dollars on partially performance-
based work assignment contracts (where some work assignments are performance-
based and some are not).
During our sample selection process, we found three contracts that were incorrectly
coded as performance-based due to data entry mistakes by contracting officials.
While we did not verify whether there were additional coding errors, EPA officials
said information entered into ICMS is not validated. EPA should ensure that PBSC
coding in ICMS is as accurate as possible.
Reporting Improvements Needed
In accordance with FAR 4.602, FPDS is designed to serve as:
" A basis for recurring and special reports to the President, the Congress, GAO,
Federal executive agencies, and the general public.
" A means of measuring and assessing the impact of Federal contracting on the
Nation s economy.
" Data for other policy and management control purposes.
Until EPA ensures that ICMS data is consistent with FPDS, the Agency s
credibility as well as any subsequent stakeholder decision making could be
adversely affected.
15
-------
Recommendations
We recommend that the Assistant Administrator for Administration and Resources
Management require the Director, Office of Acquisition Management to:
3-1. Periodically review the accuracy of the contracting data contained within
ICMS, including service codes, and make necessary corrections in a timely
manner.
3-2. Periodically compare ICMS and FPDS data to ensure the reliability of the
information, and examine any major differences and correct in a timely
manner.
3-3. Summarize the steps taken or planned, and progress made in correcting the
PBSC data quality issues in the Office of Administration and Resources
Management s Federal Manager s Financial Integrity Act report.
Agency Comments and OIG Evaluation
The Office of Administration and Resources Management agreed with the findings
and recommendations and included planned corrective actions. The response
mentions significant problems with EPA s contract tracking systems, noting that
EPA s current systems do not meet our business needs. They provide inadequate
reporting, cumbersome tracking status, and redundant data entries which result in
extraordinary inefficiencies and errors. The Office of Administration and
Resources Management anticipates that the new system it has proposed will correct
these problems. The full response from the Office of Administration and
Resources Management is included in this report as Appendix B.
16
-------
Appendix A
Details on Scope and Methodology
We performed this audit in accordance with the Government Auditing Standards, issued by the
Comptroller General of the United States. We assessed EPA s policies and procedures to ensure
performance-based service contracts were awarded when possible, determined whether contract
statements of work were designed with performance-based attributes, and determined whether
PBSC information was reported accurately. We performed this audit from December 2001
through November 2002.
To assess EPA s policy and procedures, we reviewed OMB s Office of Federal Procurement
Policy guidelines and policy letters and interviewed OMB and GAO personnel concerning
performance-based contracting. We interviewed EPA personnel at Washington, B.C.;
Cincinnati, Ohio; Research Triangle Park, North Carolina; and Philadelphia, Pennsylvania from
the following offices (the offices are in Washington unless otherwise noted):
* Office of Administration and Resources Management
" Office of Acquisition Management
" Policy, Training and Oversight Division
" Headquarters Procurement Operations Division
" Superfund/Resource Conservation and Recovery Act
Regional Procurement Operations Division
" Research Triangle Park Procurement Operations Division
" Cincinnati Procurement Operations Division
„. Office of Environmental Information
„. Office of Solid Waste and Emergency Response
„. Andrew W. Breidenbach Environmental Research Center (Cincinnati)
„. Region 3 (Philadelphia)
Using EPA s active contracts listings as of July 2001 (preliminary research) and June 30, 2002
(field work), we judgmentally selected a total of 30 contracts and task orders considering contract
value, contract type, service type, and area/program/media supported, to obtain a cross-section of
EPA PBSC-eligible service contracts. We selected 10 contracts/task orders
(approximately $16 million maximum potential value) not identified as PBSC even though
similar services had previously been identified as PBSC out of approximately 500
non-performance-based contracts ($11.7 billion maximum potential value). We also selected
20 contracts/task orders (approximately $78 million maximum potential value) out of
approximately 120 contracts ($718 million maximum potential value) that EPA identified as
PBSC.
We did not randomly select the sample because of the relatively low number of EPA
performance-based contracts (approximately 120) and the presence of many low dollar value,
similar contracts. While contract type may be a significant factor in determining whether PBSC
17
-------
contracts were designed to take advantage of the benefits, we did not specifically review the
contracts to determine whether the most appropriate contract type was used in each situation.
To assess whether EPA s policies and procedures ensured performance-based service contracts
were awarded when possible, we reviewed our sample of 10 non-performance-based contracts to
determine reasons for not using the performance-based method. We also reviewed EPA s
September 4, 2002, Acquisition Forecast Database to determine whether the performance-based
method was planned for additional forecasted acquisitions.
To determine whether contract statements of work were designed with performance-based
attributes, we reviewed the 20 statements of work for contracts and task orders to evaluate
whether they contained the required FAR performance-based attributes.
To determine whether EPA reported performance-based acquisitions accurately, we compared
summary-level FPDS information to EPA s PBSC actions recorded in EPA s ICMS. For our
sample of 20 performance-based contracts, we determined whether the action was properly
recorded as PBSC in ICMS. We also interviewed system data operators to determine reasons for
differences. However, we did not review system controls for accurate, complete, and timely data.
Prior Audit Coverage
GAO Report GAO-02-1049, Guidance Needed for Using Performance-Based Service
Contracting (September 2002), raised concerns as to whether agencies have a good
understanding of PBSC and how to take full advantage of its benefits. GAO reported that
various agency officials noted the need for better PBSC guidance and criteria on which contracts
should be called performance-based. OMB s Office of Federal Procurement Policy recognized
this problem and is developing new guidance designed to improve agency use of PBSC.
18
-------
Appendix B
^
\
Agency Response to the Draft Report
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
MAR 19 2003
MEMORANDUM
SUBJECT:
FROM:
TO:
OFFICE OF
ADMINISTRATION
AND RESOURCES
MANAGEMENT
Draft Audit Report
EPA Could Increase Savings and Improve Quality
Through Greater Use of Performance-Based Service Contracts
Assignment No. 2002-000319
Morris X. Winn lotted!
Assistant Administrator
Robert Mitchell
Director for Contracts Audit
Office of Inspector General
Thank you for the opportunity to provide comments on the subject draft audit report We
are in general agreement with the findings, and we appreciate your support in helping us increase
the use of performance-based service contracting (PBSC) to meet the Agency s goals.
As you are aware, the Administration has set Government-wide goals, which cannot be
met without Agency-wide support and participation. With this is mind, we have taken several
initiatives to promote and market PBSC throughout the Agency. These initiatives, which
strongly emphasize a sharing of the responsibility with our program office customers, include:
providing additional, more focused training upon request; providing improved PBSC outreach,
direction and tools to make PBSC more user friendly to EPA offices; providing hands-on
assistance in converting additional program requirements to PBSC; and sharing lessons learned
and best practices throughout the Agency. Again, we appreciate any assistance that the OIG can
provide in helping to obtain much-needed buy-in from senior level officials.
19
-------
Detailed comments on the report are provided below. While we generally agree with
your draft report, we take issue with several specific findings (see comments below). Our
responses are discussed by subject in the order of the appearance in the report
If you have any questions or comments, please call Judy Davis. Director, Office of
Acquisition Management, at 564-4310 or Joan Wooley at 564-4732.
Chapter 2: EPA Needs to Increase Use of Performance-Based Service Contracting
We agree that EPA needs to increase performance-based service contracting. However, the
report states that of the $599M in PBSC-eligible obligations for the 9-month period reviewed, it
was determined that EPA could have saved as much as $72.5M by better applying PBSC. The
report does not identify how the $599M figure was determined. It would be helpful to us if the
OIG explained how the $599M figure was calculated.
Page 4. Finding: EPA Could Award More Performance-Based Contracts
We agree with the overall finding; however, we disagree with the following and have provided
our rationale.
Contract 68D01051 for Cafeteria Services and Operation of EPA employee store. This contract
is a license agreement for the contractor to operate the cafeteria and employee store. A small
subsidy for the operation of the cafeteria is provided by the Government. The subsidy was
designed to decrease, based on the expectation of the contractor making a profit once the EPA
facility was fully occupied. With the decreasing subsidy and the results-oriented SOW, we
believe that the agreement is more in line with a PBSC contract. However, the nature of the
agreement does not lend itself to specific performance standards (other than those specified by
law (sanitation, business licenses, etc.). Therefore no written incentives or disincentives were
included (other than the incentive for the contractor to make enough money to pay for all
expenses). Based on the facts provided we recommend that this contract be deleted from
table 2-1.
Contract 68-W-00-128 for Voice Mail System Maintenance. We agree that this was not awarded
fully as a performance-based contract. However, we believe that the contract does contain many
performance-based elements. For example, the statement of work is written in terms of what
the output must be rather than how the work is to be accomplished, and measurable
performance standards are defined for all major task areas. The contract is scheduled for
recompete in 2004 and we have started early discussions with the program office to ensure that
the follow-on contract is awarded as full performance-based.
Contract 68-W-00-125 Technical Support for Compliance Assurance. We agree that some tasks
under this contract could be performance-based. However, at the time the contract was awarded,
based on the uncertainties associated with the level of effort and other variables, it would have
been difficult to convert the statement of work (SOW) to PBSC. The contract expires
20
-------
September 2003, and we have been advised by the program office that there will be no follow-on
contract for these services. We will however ensure that where feasible, task orders are PBSC.
Page 7. Finding: Contracts That Were Prescriptive
In our efforts to transition to PBSC some of our contracts may have been prescriptive; however,
we disagree with the following:
Contract 68-C-00-189 and Contract D-98-169 for Security Guard Services. We disagree with the
findings and recommend that these contracts be deleted from table 2-3 as being contracts that
were prescriptive. Since the events of September 11th we had to modify our guard services
contracts to clearly specify our requirements. We no longer consider these services to be low
risk, nor do we believe that the requirements restrict the contractor from using innovative
approaches or shifting responsibility.
Contract 68-W-02-020 Property Management Services. We disagree that this contract contained
PBSC attributes which were overly prescriptive and recommend that this contract be deleted
from the list of contracts identified as being prescriptive. The SOW included a brief background
and purpose, then provided generalized support requirements, further delineated by specific
performance objectives/appropriate performance measures/performance standards. Although we
included a description of the labor categories, we do not believe that this impedes the
contractor s ability to offer innovative approaches to determine how best to perform.
Page 7/8. Finding: Incentives and Disincentives not Meaningful.
Contract 68-W-00-104 Security Support Services. We agree with the OIG that this contract,
awarded in June 2000, contains numerous disincentives of nominal dollar value. Working with
the program office, we will review all of the incentives and disincentives in this contract and
revise them as appropriate to be more meaningful. By June 30, 2003, we will ensure that the
revised incentives and disincentives are incorporated into the contract for option period two.
Page 14 Finding: Performance-Based Task Orders Issued Under Non-Performance-Based
Contracts Not Shown as PBSC in FPDS.
We agree that some data was not properly coded as PBSC or non-PBSC. Corrections have now
been made to ICMS data, based on a review of ICMS data by OAM s operations divisions.
Those changes have been re-submitted to FPDS and have been processed. We have not yet seen
the results.
We also agree that PBSC task orders under non-PBSC contracts were not being submitted as
PBSC to FPDS. This discrepancy accounted for mostof the under-reporting of PBSC actions to
FPDS. EPA has corrected its reporting process and has resubmitted the PBSC task order actions
to FPDS. Due to a six week shutdown of FPDS, the re-submission has just been processed.
Again, we have not yet seen the results.
21
-------
Page 15 Finding: Reporting Improvements Needed.
The report states: Until EPA ensures that ICMS data is consistent with FPDS, the Agency s
credibility as well as any subsequent stakeholder decision making could be adversely affected.
ICMS data changes daily, while FPDS is currently updated only quarterly. One of the E-Gov
initiatives planned for FY2004 is development of a new and better data gathering system, Federal
Procurement Data System - Next Generation (FPDS-NG). This system will be updated more
frequently - possibly overnight, but a final decision has not been made. After FPDS-NG is
implemented, it is more likely that the two systems will more closely match.
It should also be noted that EPA s current systems do not meet our business needs. They provide
inadequate reporting, cumbersome tracking status, and redundant data entries which result in
extraordinary inefficiencies and errors. We need a system that can interface with Government-
wide systems as described in the ACQUISITION SYSTEM Project proposed in the FY 2005
OAM Capital Planning for Investment Control Process (CPIC).
The proposed system will be a secure web-based tool, eliminate duplicate data entry, track actual
costs and progress in real time, and provide a Commercial Off The Shelf (COTS) application
consistent with the President s E-Gov initiatives. We appreciate the OIG s help in soliciting
support for this system.
RECOMMENDATIONS
The OIG recommends that the Assistant Administrator for Administration and Resources
Management, in coordination with the Senior Resource Officials for each Headquarters
and regional office and other field locations promote maximum use of PBSC by:
Recommendation 2-1
OIG: Developing and implementing a PBSC plan that includes short-term and long-term
strategies for maximizing the use of performance based contracting. The plan should establish
goals for PBSC use in program offices and monitor progress against those goals.
OARM s Response: We concur with this recommendation and will provide a plan of action to
comply with the recommendation in the final report. It would be helpful if the IG would assist us
in soliciting support from senior officials in the Agency to engage AAs and RAs in helping the
Agency meet its goals.
Recommendation 2-2
OIG: Developing and implementing procedures to improve the use of PBSC, such as deploying
experienced PBSC contracting personnel to assist program offices in developing performance-
based statements of work, and ensuring that the statements of work contain the required
elements.
22
-------
OARM s Response: Although we concur with this recommendation, we currently provide
experienced PBSC contracting personnel to assist program offices, develop PBSC SOW
templates, etc. We will provide details in the final report.
Recommendation 2-3
OIG: Identifying forums, such as the Innovation Action Council, to help further promote PBSC,
and to discuss PBSC best practices and issues on a regular basis.
OARM s Response: We concur with the recommendation. We currently engage in several
venues to promote PBSC. We solicit the IG support in getting buy-in at the senior level.
Recommendation 2-4
OIG: Developing and implementing a procurement policy addressing the FAR Subpart 7.105
requirement to implement performance-based contracting methods, or provide a rationale for not
using those methods.
OARM s Response: We concur with this recommendation.
The OIG recommend that the Assistant Administrator for Administration and Resources
Management require the Director, Office of Acquisition Management to:
Recommendation 3-1
OIG: Periodically review the accuracy of the contracting data contained within ICMS, including
service codes, and make necessary corrections in a timely manner.
OARM s Response: We concur with this recommendation. Please see comments referenced
under the findings on page 14.
Recommendation 3-2
OIG: Periodically compare ICMS and FPDS data to ensure the reliability of the information, and
examine any major differences and correct in a timely manner.
OARM s Response: We concur with this recommendation.
Recommendation 3-3
OIG: Summarize the steps taken or planned, and progress made in correcting the PBSC data
quality issues in OARM s Federal Manager s Financial Integrity Act report.
OARM s Response: We concur with this recommendation and will address this and other data
quality issues in the Federal Manager s Financial Integrity Act report.
23
-------
24
-------
Appendix C
Report Distribution
Assistant Administrator for Administration and Resources Management
Assistant Administrator for Environmental Information
Assistant Administrator for Solid Waste and Emergency Response
Assistant Administrator for Air and Radiation
Assistant Administrator for Water
Assistant Administrator for Prevention Pesticides and Toxic Substances,
Assistant Administrator for International Affairs
Assistant Administrator for Research and Development
Agency Followup Official
Agency Audit Followup Coordinator
Associate Administrator for Congressional and Intergovernmental Relations
Associate Administrator for Public Affairs
Director, Office of Acquisition Management
Director, Office of Emergency and Remedial Response
Audit Liaison, Office of Administration and Resources Management
Audit Liaison, Office of Environmental Information
Audit Liaison, Office of Solid Waste and Emergency Response
Inspector General (2410)
Regional Administrators, Regions 1 through 10
Regional Audit Followup Coordinators, Regions 1 through 10
25
------- |