\saA  OIG
 % PPO^   OFFICE OF INSPECTOR GENERAL
                          Catalyst for Improving the Environment
Audit Report
        EPA Could Increase Savings and
        Improve Quality Through Greater Use of
        Performance-Based Service Contracts
        Report No. 2003-P-00008
        March 31, 2003

-------
Report Contributors:                   Herbert Maletz
                                      Denton Stafford
                                      Charles Watts
                                      Brian Lick
Abbreviations
EPA         Environmental Protection Agency
FAR         Federal Acquisition Regulation
FPDS        Federal Procurement Data System
GAO         General Accounting Office
ICMS        Integrated Contracts Management System
OIG         Office of Inspector General
OMB        Office of Management and Budget
PBSC        Performance-Based Service Contracting
START      Superfund Technical Assessment and Response Team

-------
         v.      UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
         '.                    WASHINGTON, DC  20460

   '"(
                                                                        OFFICE OF
                                                                    INSPECTOR GENERAL

                                    March 31,2003

MEMORANDUM

SUBJECT:        Audit Report:
                  EPA Could Increase Savings and Improve Quality
                  Through Greater Use of Performance-Based Service Contracts
                  Report No. 2003-P-00008
FROM:           Robert Mitchell
                  Director for Contract Audits

TO:               Morris X. Winn
                  Assistant Administrator
                  Office of Administration and Resources Management
This is our final report on the subject audit conducted by the Office of Inspector General (OIG)
of the U.S. Environmental Protection Agency (EPA). This audit report contains findings that
describe the problems the OIG has identified and corrective actions the  OIG recommends.  This
audit report represents the opinion of the OIG and the findings contained in this report do not
necessarily represent the final EPA position. Final determinations on matters in this audit report
will be made by EPA managers in accordance with established audit resolution procedures.

Action Required

In accordance with EPA Order 2750, you are required to provide a written response to the
findings and recommendations presented in this draft report within 90 calendar days of the date
of this report. You should include a corrective actions plan for agreed-upon actions, including
milestone dates.  We have no objection to the further release of this report to the public.  For your
convenience, this report will be available at http://www.epa.gov/oig/eroom.htm.

If you or your staff have any questions regarding this report, please contact me at (202) 566-0891
or Herbert Maletz at (212) 637-3058.

-------

-------
                     Executive Summary
Purpose

             Performance-Based Service Contracting (PBSC)is an acquisition methodology in
             which the Government pays for results rather than effort or process. Office of
             Management and Budget (OMB) studies have shown that PBSC saves money and
             improves contractor performance. While PBSC has existed for many years, it has
             not been widely adopted throughout the Federal Government.

             We conducted this audit to determine whether the Environmental Protection
             Agency (EPA) effectively implemented PBSC to the maximum extent possible.
             Our specific objectives were to determine whether EPA was:

             "  Awarding performance-based service contracts whenever possible and
                designing the contracts to take full advantage of the benefits.

             "  Reporting accurate PBSC information to OMB.

Results of Review

             EPA has made limited use of PBSC.  Generally, EPA has only used PBSC to
             obtain commercially available services, such as janitorial and landscaping work.
             We found that EPA awarded contracts that were not performance-based for
             services previously awarded as performance-based. Also, many contracts EPA
             identified as performance-based were not designed to take advantage of PBSC s
             benefits. These contracts were too prescriptive in how the work was to be carried
             out or did not provide meaningful incentives or disincentives. Based on
             discussions with Agency officials, this occurred because EPA did not adequately
             plan or hold officials accountable for increasing PBSC use, and there was a
             general reluctance to shift the responsibility for outcomes to contractors. As a
             result, EPA missed opportunities to achieve greater cost savings and improve
             contractor performance. Of the $599 million in PBSC-eligible obligations for the
             9-month period reviewed, we determined that EPA could have saved as much as
             $72.5 million by better applying PBSC.

             Also, OMB s Federal Procurement Data System (FPDS) showed nearly 50 percent
             less PBSC obligations than EPA reported in its own Integrated Contracts
             Management System (ICMS). This primarily occurred because FPDS did not
             show performance-based task orders issued under non-performance-based
             contracts as PBSC; EPA made the necessary corrections in ICMS but had not
             submitted them to FPDS. In addition, EPA officials said ICMS lacked  the
             capability to provide all data needed in FPDS. As a result, EPA s contracting

-------
             actions were inaccurately portrayed in the national database, which serves as a
             basis for reporting to Congress, the President, and the general public; and for
             performing analysis.

Recommendations

             We recommend that the Assistant Administrator for Administration and
             Resources Management take certain actions to improve accountability and
             procedures in relation to PBSC.  We also recommend that the Assistant
             Administrator take actions so that the accuracy of ICMS is reviewed and that
             ICMS and FPDS data is periodically compared.

Agency Comments and OIG Evaluation

             On February 14, 2003, we issued a draft report to the Office of Administration
             and Resources Management, and received a response on March 19, 2003
             (see Appendix B). The Office generally agreed with the findings and
             recommendations and included planned corrective actions. The Office
             acknowledged significant problems with contract tracking systems, and noted a
             new system it has proposed will correct the problems.  In its response, the Office
             also provided additional information, and we clarified our final report as needed.

-------
                  Table  of Contents
Executive Summary
 Chapters
     1    Introduction 	  1





     2    EPA Needs to Increase Use of PBSC	  3





     3    EPA Needs to Improve PBSC Reporting 	  13
 Appendices




     A   Details on Scope and Methodology	  17




     B   Agency Response to the Draft Report	  19




     C   Report Distribution  	  25

-------

-------
                                Chapter 1
                                 Introduction
Purpose
             Performance-Based Service Contracting (PBSC) is an acquisition methodology in
             which the Government pays for results rather than effort or process.  Office of
             Management and Budget (OMB) studies have shown that PBSC saves money and
             improves contractor performance. While PBSC has existed for many years, it has
             not been widely adopted throughout the Federal Government.

             We conducted this audit to determine whether the Environmental Protection
             Agency (EPA) effectively implemented PBSC to the maximum extent possible.
             Our specific objectives were to determine whether EPA was:

              "  Awarding performance-based service contracts whenever possible and
                designing the contracts to take full advantage of the benefits.
              "  Reporting accurate PBSC information to OMB.
Background
             PBSC allows contractors to propose the most innovative, effective, efficient, and
             economical approach their business experience and ability will allow. Such
             contracts can help an agency avoid cost overruns, schedule delays, and failure to
             achieve specified results.  EPA participated in OMB s PBSC studies as well as its
             own pilot projects and has experienced these benefits.

             OMB s Office of Federal  Procurement Policy has encouraged greater use of PBSC
             for over 10 years. In April 1991, this Office issued Policy Letter 91-2, Service
             Contracting.  This Policy Letter defined PBSC, required the use of PBSC where
             practicable, and required agencies to better match their acquisition and contract
             administration strategies to the specific requirements.

             Policy Letter 91-2 was incorporated into the Federal Acquisition Regulation
             (FAR) in 1997. Subpart 37.1 of the FAR, Service Contracts-General, states when
             acquiring services, agencies must use performance-based contracting methods to
             the maximum extent practicable, except for architect-engineer services,
             construction, utility services, and services incidental to supply purchases.

-------
             FAR Subpart 37.6, Performance-Based Contracting, first issued in 1997,
             prescribes policies and procedures for PBSC. The FAR notes performance-based
             contracts:

              "  Describe the requirements in terms of results required rather than the methods
                of performing the work;
              "  Use measurable performance standards (i.e., terms of quality, timeliness,
                quantity, etc.) and quality assurance surveillance plans;
              "  Specify procedures for reductions of fee or for reductions to the price of a
                fixed-price contract when services are not performed or do not meet contract
                requirements; and
              "  Include performance incentives where appropriate.

             OMB s Office of Federal Procurement Policy issued Information on Best
             Practices for Performance-Based Service Contracting in October 1998. This
             guide notes studies have shown that PBSC has the potential to improve customer
             satisfaction by more than 18 percent and achieve average contract cost savings of
             15 percent.  However, the Office noted that application of only selected aspects of
             the PBSC methodology can limit benefits.

             Agencies must report PBSC information to the Federal Procurement Data System,
             which is the central repository of statistical information on Federal contracting.
             The system contains detailed information on contract actions over $25,000, and
             summary data on procurements less than  $25,000.
Scope and Methodology
             From October 2001 through June 2002, EPA acquired approximately
             $599 million in services that were PBSC-eligible. This information was obtained
             from the November 7, 2002, Integrated Contracts Management System (ICMS).
             We performed this audit in accordance with Government Auditing Standards,
             issued by the Comptroller General of the United States.  We judgmentally selected
             a sample of 30 contracts and task orders for review.  We assessed EPA s policies
             and procedures to ensure performance-based contracts were awarded when
             possible, determined whether contracts statements of work were designed with
             performance-based attributes, and determined whether PBSC information was
             reported accurately.  During the audit, we interviewed numerous personnel. We
             performed this audit from December 2001 through November 2002. As part of
             our review, we took into account a prior General Accounting Office (GAO) report
             (GAO-02-1049, Guidance Needed for Using Performance-Based Service
             Contracting, September 2002).  Details on our scope and methodology are in
             Appendix A.

-------
                                Chapter  2

              EPA Needs  to  Increase Use of PBSC

             EPA has made limited use of PBSC.  Generally, EPA has used PBSC to obtain
             commercially available services, such as janitorial and landscaping work.  We
             found that EPA awarded non-performance-based contracts for services previously
             awarded as performance-based. Also, many contracts EPA identified as
             performance-based were not designed to take advantage of PBSC s benefits.
             These contracts were too prescriptive in how the work was to be carried out or did
             not provide meaningful incentives or disincentives. Based on discussions with
             Agency officials, these conditions occurred because EPA did not adequately plan
             or hold officials accountable for increasing PBSC use, and there was a general
             reluctance to shift the responsibility for outcomes to contractors. As a result,  EPA
             missed opportunities to achieve greater cost savings and improve contractor
             performance. Of the $599 million in PBSC-eligible obligations for the 9-month
             period ended June 30, 2002, we determined that EPA could have saved as much
             as $72.5 million by better applying PBSC.

OMB  Has Encouraged Greater Use of PBSC Since 1991

             As noted in Chapter 1, OMB has required the use of PBSC where practicable
             since 1991. In 1997, this requirement was incorporated into the FAR,
             Subpart 37.6, Performance-Based Contracting.  PBSC is intended to ensure that
             required performance quality levels are achieved and that total payment is related
             to the degree that services performed meet contract standards. Performance-based
             contracts describe the desired end result (outcomes) expected of the contractor,
             but do not direct how the work is to be performed.

             In April 2002, FAR Subpart 7.105, Contents of Written Acquisition Plans,
             required that acquisition planners develop written acquisition plans for service
             contracts or task orders that describe strategies for implementing performance-
             based contracting methods, or provide rationale for not using those methods.
             EPA s Contracts Management Manual, Chapter 1 (February 1999) notes EPA s
             Senior Resource Officials are the primary acquisition planners.

EPA Has Made Limited  Use of PBSC

             Generally, EPA has limited PBSC to services performed widely in the commercial
             sector: custodial services, building maintenance, or landscaping. As noted in a
             GAO report, this was the case with many agencies. These  services easily lend
             themselves to PBSC because measuring and specifying outcomes are relatively
             straightforward. Additionally, these generally non-complex services do not
             present substantial risk to the Government.

-------
             OMB has set Government-wide goals for PBSC. Beginning in fiscal 2001,
             10 percent of all eligible service contracting obligations were to be performance-
             based. The PBSC goals incrementally increase 10 percent annually until 2005.
             The goal for fiscal 2002 was 20 percent.

             For fiscal 2001, EPA reported approximately 11 percent of its eligible contract
             obligations were performance-based, and thus met its goal of 10 percent.
             However, for fiscal 2002, EPA only reported that 16.6 percent of contract
             obligations were performance-based, short of OMB  s goal of 20 percent. EPA
             needs to increase its effort to meet OMB s PBSC goals.

EPA Could Award More Performance-Based Contracts
             Between 1998 and 2001, EPA awarded many contracts without employing PBSC
             attributes, for services similar to previously awarded EPA performance-based
             contracts. We reviewed 10 such contracts (with a maximum potential value of
             $16.3 million), as shown in Table 2-1:

                Table 2-1:  Contracts Not Performance-Based for Which Similar Services Were
                              Previously Awarded as Performance-Based
Contract Maximum
Number Service Potential Value
68-D-01-051
68-W-00-128
68-W-00-125
68-W-99-078
68-D-99-026
68-D-99-014
68-W-99-012
68-D-98-166
68-D-98-092
68-D-98-029
Total
Cafeteria Services and Operation of EPA
Employee Store
Voice Mail System Maintenance
Technical Supportfor Compliance Assurance
Facility Operatbn and Maintenance Service for trie
Manchester EPA Laboratory
Landscape/Grounds Maintenance Support
National Performance Audit Program
(Air Monitoring)
Perimeter Guard Services
Security Guard Services
Peer Review, Workshops, Training Sessions, and
Conference Support
Maintenance Services for the Natbnal Airand
Radiation Laboratory
$264,000
677,000
2,813,000
916,000
162,000
7,127,000
3,218,000
505,000
292,000
284,000
$16,258,000
             Most of these contracts were for services widely performed in the commercial
             sector, and did not present substantial risk to the Agency. Measuring and
             specifying outcomes for these types of services were relatively straightforward.
             Based on the types of services acquired and the fact that EPA had previously
             awarded a performance-based contract of a similar nature, we believe these
             contracts could have been performance-based.

-------
             EPA Office of Acquisition Management officials said the above contracts were
             not performance-based for a variety of reasons, including urgency of need,
             inability to define performance objectives, and inexperience in using PBSC
             methods (some earlier contracts).  The officials also stated the Voice Mail System
             Maintenance contract was not fully performance-based primarily because staff did
             not have enough time to convert the statement of work.

             In addition to the 10 non-performance-based contract files reviewed, we also
             selectively reviewed ICMS data as of June 30, 2002, and found instances where
             EPA awarded non-performance-based contracts even though contracts for similar
             services were previously performance-based:

              "  ICMS reported four non-performance-based facilities operation support
                services contracts, even though at least seven other facilities operation support
                services contracts within the past 5 years were performance-based.

              "  Two landscaping contracts were not identified as performance-based, even
                though in the last 2 years EPA awarded at least two other landscaping
                contracts identified as  performance-based.

              "  EPA awarded a janitorial/custodial contract not identified as performance
                based, even though within the last 5 years EPA awarded at least nine other
                performance-based janitorial/custodial contracts.

              "  EPA awarded a non-performance-based mail service  contract, even though
                within the last 5 years  EPA has awarded at least five  other performance-based
                mail services contracts.

Performance-Based Contracts Not Always Designed to Take
Advantage of Benefits

             Performance-based contracts were not always designed to take advantage of
             PBSC benefits.  We reviewed 20 individual performance-based contracts and task
             orders ($77.8 million maximum potential value), which generally contained at
             least one of the following  performance-based attributes required by FAR,
             Subpart 37.6:

              "  Performance requirements described in terms of results required.
              "  Measurable performance standards and quality assurance surveillance plans.
              "  Financial disincentives.
              "  Performance incentives.

             Eight of the contract actions appeared to generally be designed to take advantage
             of PBSC benefits, even though they may not have contained all four of the above

-------
PBSC attributes.  However, in our opinion, the other 12 contracts were not
designed to take advantage of PBSC s benefits. These contracts  PBSC attributes
were either prescriptive in how the work was to be carried out or did not provide
meaningful incentives to increase contractor efficiency or penalize contractors for
poor performance.

Performance-Based Contracts Designed to Take Advantage of PBSC
Benefits

The following eight performance-based contracts appeared to generally be
designed to take advantage of PBSC benefits (see Table 2-2).  These contracts
exhibited many of the basic PBSC attributes.
Table 2-2:
Performance-Based Contracts Designed to Take Advantage of PBSC Benefits
Contract
Number
68-C-01-025
68-D-01-049
68-S3-00-02
Work Assmnt. #
SE3-02-03-010
68-W-00-075
68-W-00-095
68-W-01-018
Task Order # 005
68-W-01-059
68-W-02-020
Total
Maximum
Service Potential Value
Study to Determine Concentrations of
Pollutants in Waste Water
Janitorial Services for Facilities
START (Superfund Technical Assessment and
Response Team) Removal Site Evaluation
Contract Laboratory Program-Analytical
Support
Energy Star Hotline and Distribution Center
ESAT (Environmental Services Assistance
Team) Performance Evaluation and Field
Support Services
Warehousing and Storage Services
Property Management Services
$48,500
732,000
39,000
2,245,000
2,982,000
8,148,000
13,557,000
2,117,000
$29,868,500
The START Removal Site Evaluation contract action illustrates how PBSC can
provide benefits. EPA only specified the desired end result, not the process to be
used.  EPA obtained two competing proposals and found both to be technically
sound. However, while one firm proposed using standard sampling technology,
the other firm proposed using a new technology that had proven successful at
another site. The proposals differed significantly in estimated costs: the estimate
for the standard sampling technology was $154,000 while the new technology
estimate was $39,000, or $115,000 less. EPA selected the lower cost proposal.
By not being prescriptive, EPA opened the door to more innovative approaches
that cost less and possibly improved service.

-------
While the eight contract actions in Table 2-2 represent EPA s best PBSC efforts,
it is not our intent to suggest that the attributes could not be improved.
Contracts That Were Prescriptive

Seven contracts/actions, although identified as performance-based, were
prescriptive in how the work was to be performed, and frequently required
direction from EPA project officers (see Table 2-3). Most of these contracts were
for traditional, relatively low-risk services.

                  Table 2-3: Contracts That Were Prescriptive
Contract Maximum
Number Service Potential Value
68-C-00-137
68-C-00-189
68-D-98-169
68-D-00-200
68-S3-99-04
Task Order # 12
68-W-99-082
68-W-01-002
Total
Study to Estimate Selected Chemical Levels in
Fish Tissue
Security Guard Services
Security Guard Services
Facility Maintenance and Equipment Operation
of Lab
ERRS (Emergency Rapid Response Service)
Contamination Removal
Administrative Support Services
Information Management Support Services
$13,800,000
3,000,000
774,000
4,600,000
1,900,000
2,300,000
343,000
$26,717,000
On the surface, these contracts appeared to contain the PBSC attributes.
However, a closer review indicated prescriptive statements of work that might
have impeded the contractor s innovation and freedom to decide how best to
perform the work.  For example, one guard services contract (68-C-00-189)
prescribed the number of guards desired, specific posts to be manned, specific
sizes for uniform patches and vehicle lettering, and specific age of all guard
vehicles. These examples illustrate EPA s reluctance to  allow contractors the
flexibility to use innovative approaches or shift responsibility for outcomes.

Incentives and Disincentives Not Meaningful

Five contracts/actions, which EPA identified as performance-based, did not have
meaningful incentives and/or disincentives. Although two provided negative
incentives, the proposed penalties were low dollar amounts. The other three did
not include any incentives (see Table 2-4).

-------
            Table 2-4: Incentives/Lack of Incentives Not Meaningful
Contract Maximum
Number Service Potential Value
68-C-99-252
68-C-99-254
68-W-00-104
68-W7-0055
Task Order #408
68-W7-0055
Task Order #422
Total
Environmental Impact Analysis
Administrative Support Services
Security Support Services
Library Systems Support
Operations of National Computer Center
$1,996,000
2,573,000
11,413,000
101,000
5,106,000
$21,189,000
For example, while the administrative support services contract (68-C-99-254)
exhibited PBSC attributes, its negative incentives were not meaningful.  The tasks
were organized in such a way that minuscule penalties could be assessed.
Specifically, the penalty assessed against poor performance for inventorying the
stockroom was 0.4 percent of $4,004.33 per month, or $16.02.

Superfund PBSC

While most of EPA s current performance-based contracts are for less complex
and lower risk services that are widely available in the commercial sector, its
largest contracts have been more complex, higher risk, cost reimbursable, level of
effort types, especially for the Superfund Program. These contracts place more of
the risk for cost and performance on the Government; contractors are only
required to give their best effort, and all reasonable and allowable costs are
reimbursed.

The Office of Acquisition Management and Office of Emergency and Remedial
Response developed PBSC model statements of work templates for various
Emergency Rapid Response Service, START, and  Response Action Contract
assignments  and tasks that take place on a regular basis.  According to Office of
Emergency and Remedial Response officials, the overall structure of these
multi-year time and materials contracts was not developed using the PBSC
methodology, and the use of these templates was optional. These templates were
developed in 2002 as a temporary measure to increase PBSC use under the current
contracts, and were not widely used at the time of our review.  Further, these
templates by design did not significantly change current contract management
practices of relying upon close EPA contractor oversight. Therefore, until the
basic contract incorporates all PBSC features and is managed accordingly, EPA
may not maximize PBSC benefits.

-------
More PBSC Planning and Accountability Needed

             To promote the use of PBSC, EPA s Office of Acquisition Management relied on
             a marketing strategy involving training and assistance to program offices.
             However, EPA did not adequately plan or hold responsible officials accountable
             for increasing PBSC s use.  We believe this is primarily due to Agency officials
             reluctance to shift the responsibility for outcomes to contractors.

             Current PBSC Planning Not Specific Enough

             To meet the PBSC goals, the Office of Acquisition Management developed a
             short-term plan for fiscal 2002, which focused on:

              " Sharing responsibility for meeting the goal with program office customers;
              " Implementing additional, more focused PBSC training;
              " Providing improved PBSC outreach, direction, and tools to make PBSC more
                 user friendly  to EPA offices;
              " Converting additional program requirements to PBSC; and, ultimately,
              " Sharing lessons learned and best practices throughout the Agency.

             This plan generally emphasized greater PBSC use throughout EPA, but there was
             no high-level accountability to ensure the plan s implementation. The plan did
             not outline a detailed strategy to identify additional program requirements that
             could be converted to PBSC, although such a strategy is planned for 2003.
             Furthermore, since EPA contracts generally last 3 years or longer, EPA s plan
             should also include a long-term PBSC strategy to address contracts that will not
             need to be replaced for some time.

             EPA s Acquisition Forecast Database could help identify planned performance-
             based contracts. This web site provides the business community with information
             about upcoming EPA procurements. As of September 4, 2002, of the 204 planned
             contracts listed, 38 (19 percent) were identified as performance-based.  Our
             review of the list showed that some traditional services, such as library operations,
             were not identified as performance-based.

             Lack of Accountability For PBSC Within EPA

             EPA s Office of Acquisition Management has attempted to market and promote
             PBSC, but has experienced difficulties with obtaining firm program office
             commitments on future performance-based acquisitions. The Office s staff said
             this is due to the general reluctance of program officials to change their current
             culture of managing large, cost-reimbursable, level-of-effort-type contracts.
             Program officials echoed this sentiment, expressing concerns about major changes
             and the belief that they would be losing control of their business.  While the
             Office of Acquisition Management is primarily responsible for all EPA

-------
             contracting activity, the Agency has not held program officials accountable to
             ensure the adoption and implementation of PBSC agency-wide. For example,
             another agency (the U.S. State Department) issued a directive that all new service
             contracts must be performance-based, with any deviations justified in writing and
             approved.  EPA may need such a commitment from their Senior Resource
             Officials, the primary acquisition planners.

EPA Pilot Project Exhibited  Benefits

             EPA was part of OMB  s PBSC pilot project from 1994 to 1998 to encourage
             Government-wide acceptance of PBSC and to evaluate its effects. EPA
             participated with two contracts   one for operation of a telephone hotline and the
             other for training. The  contract price was reported to decrease by 31 percent for
             the hotline and 6 percent for training. In Fiscal 1997, with assistance from OMB,
             EPA conducted a PBSC study within the Superfund program.  Nine projects were
             involved in this study, and most appeared to save money. For example, at the
             Oronogo-Duenweg Mining  Site, PBSC was responsible for lowering the project
             costs by 30-35 percent while improving the quality of work.

Significant Savings Could Be Achieved

             During our review, we found that EPA missed opportunities to achieve greater
             cost savings and higher quality service. For the 9-month period reviewed
             (October 1, 2001, through June 30, 2002), EPA reported in ICMS (as of
             November 7, 2002) that it obligated a total of $599 million for eligible
             performance-based services, with $116 million labeled performance-based.  OMB
             studies have shown that PBSC has the potential to  improve customer satisfaction
             by more than  18 percent, and achieve average contract cost savings of 15 percent.
             Applying this potential savings to EPA s remaining obligations eligible to be
             performance-based ($483 million)  indicates that EPA could have saved as much
             as $72.5 million during this 9-month period while improving the quality of
             services obtained.

Recommendations

             We recommend that the Assistant Administrator for Administration and
             Resources Management, in  coordination with the Senior Resource Officials for
             each Headquarters and regional office and other field locations, promote
             maximum use of PBSC by:

              2-1. Developing and implementing a PBSC plan that includes short-term and
                   long-term strategies  for maximizing the use of performance based
                   contracting.  The plan should establish goals for PBSC use in program
                   offices and monitor progress against those goals.
                                         10

-------
              2-2.  Developing and implementing procedures to improve the use of PBSC,
                   such as deploying experienced PBSC contracting personnel to assist
                   program offices in developing performance-based statements of work, and
                   ensuring that the statements of work contain the required elements.

              2-3.  Identifying forums, such as the Innovation Action Council, to help further
                   promote PBSC, and to discuss PBSC best practices and issues on a regular
                   basis.

              2-4.  Developing and implementing a procurement policy addressing the FAR
                   Subpart 7.105 requirement to implement performance-based contracting
                   methods, or provide a rationale for not using those methods.

Agency Comments and OIG Evaluation

             The Office of Administration and Resources Management agreed with the
             findings and recommendations and included planned corrective actions. The
             Office also provided additional information, and we clarified the report as needed.
             The full response is included in this report as Appendix B.
                                         11

-------
12

-------
                                Chapter  3
            EPA Needs to Improve PBSC Reporting
            OMB s Federal Procurement Data System (FPDS) showed nearly 50 percent less
            PBSC obligations than EPA reported in its own ICMS. This primarily occurred
            because FPDS did not show performance-based task orders issued under non-
            performance-based contracts as being performance-based.  EPA made the
            necessary corrections in ICMS but had not submitted them to FPDS. In addition,
            EPA officials said ICMS lacked the capability to provide all data needed in FPDS.
            As a result, EPA  s contracting actions were inaccurately portrayed in the national
            database, which serves as a basis for reporting to Congress, the President, and the
            general public, and for performing analysis.
Requirements
            Public Law 93-400, Office of Federal Procurement Policy Act, requires the Office
            of Federal Procurement Policy to establish a system for collecting, evaluating, and
            disseminating Federal contract information.  FPDS is that central repository.
            FAR Subpart 4.6, Contract Reporting, requires each executive agency to establish
            and maintain for a period of 5 years a computer file,  by fiscal year, containing
            unclassified records of all procurements exceeding $25,000.  Agencies must
            transmit this information to FPDS.

            The FPDS Reporting Manual outlines PBSC information reporting requirements,
            and states that PBSC applies only to specific service codes.  Any procurement
            action coded as a product or as any of the following services is not eligible to be
            categorized as PBSC in FPDS:

            " Architect-engineer services.
            " Construction.
            " Utility services.
            " Services that are incidental to supply purchases.

            FAR Subpart 37.102 states agencies are not required to use performance-based
            contracting methods for these services. For PBSC reporting purposes, these
            services are considered  not PBSC eligible.

            ICMS is EPA s automated acquisition system and contains information on EPA s
            contracting activities. EPA converts ICMS information to a format compatible
            with FPDS, and this information is then sent to FPDS quarterly.  To report accurate
            PBSC information, the Agency must determine whether the procurement action
                                        13

-------
           meets PBSC eligibility requirements, contains required attributes, and uses an
           appropriate service code.

EPA Under-Reported PBSC Activity

           Our comparison of ICMS and FPDS summary level data for October 1, 2001, to
           June 30, 2002, disclosed that almost 50 percent of EPA s PBSC information was
           not shown as such in FPDS (see Table 3-1). EPA s total obligations in the two
           systems, as well as its PBSC-eligible obligations, were within 10 percent of each
           other.
Table 3-1: Comparison of ICMS and FPDS Data ($ in millions)
EPA Obligations
Total Obligations
Not PBSC Eligible
PBSC Eligible
PBSC
ICMS
$649
50
599
$116
FPDS
$594
45
549
$61
Percent
Under-Reported
8.5%
10.0%
8.0%
47.5%
           The PBSC  category are obligations for procurement actions that EPA has
           determined met PBSC requirements.

Performance-Based Task Orders Issued Under Non-Performance-
Based Contracts Not Shown as PBSC in FPDS

           FPDS reported $55 million less PBSC obligations than ICMS. ICMS shows
           performance-based task orders issued under non-performance-based contracts, but
           these actions were not shown in FPDS as PBSC. ICMS reported approximately
           $43 million in performance-based task order obligations issued under non-
           performance-based contracts that were not shown as PBSC in FPDS. For example,
           ICMS reported that a national telecommunication and computer services contract
           (68-W7-0055) contained approximately $29 million of performance-based task
           orders, but that amount was not shown as such in FPDS.

           EPA Office of Acquisition Management officials acknowledged ICMS and FPDS
           inconsistencies in reporting PBSC information, and stated that ICMS information
           as of June 30, 2002, which was sent to FPDS, contained these errors. System users
           subsequently corrected the ICMS information, but EPA had not submitted these
           corrections to FPDS at the time of our review. These EPA officials said system
           users data entry is normally not reviewed for accuracy.  The Office of Acquisition
           Management s fiscal 2002 Assurance Letter under the Federal Manager s Financial
           Integrity Act indicated that data quality is a major concern in EPA, and they are
           currently investigating these problems and are committed to their resolution.
                                        14

-------
Other PBSC Reporting Inconsistencies

            Another FPDS database reporting inconsistency concerns work assignments issued
            under EPA contracts. EPA designed work assignments to control work flow but,
            unlike task orders, work assignments are not separate funding instruments from the
            basic contract. Therefore, FPDS does not recognize work assignments.
            Consequently, EPA performance-based obligations organized under work
            assignments in ICMS were not reported in FPDS as a performance-based
            obligation.

            Further, work assignments issued under architectural and engineering services
            contracts are not eligible in FPDS to be performance-based due to the service code
            of the basic contract. The architectural and engineering Response Action Contract
            for EPA Region 5 included two performance-based work assignments with a
            combined value of $1.3 million. While these obligations were shown in ICMS as
            performance-based, they were not reported in FPDS as being performance-based.
            The Office of Acquisition Management s fiscal 2002 Assurance Letter notes ICMS
            cannot be used to accurately identify and report dollars on partially performance-
            based work assignment contracts (where some work assignments are performance-
            based and some are not).

            During our sample selection process, we found three contracts that were incorrectly
            coded as performance-based due to data entry mistakes by contracting officials.
            While we did not verify whether there were additional coding errors, EPA officials
            said information entered into ICMS is not validated. EPA should ensure that PBSC
            coding in ICMS is as accurate as possible.

Reporting Improvements Needed

            In accordance with FAR 4.602, FPDS is designed to serve as:

              "  A basis  for recurring and special reports to the President, the Congress, GAO,
                Federal  executive agencies, and the general public.
              "  A means of measuring and assessing the impact of Federal contracting on the
                Nation s economy.
              "  Data for other policy and management control purposes.

            Until EPA ensures that ICMS data is consistent with FPDS, the Agency s
            credibility as well as any subsequent stakeholder decision making could be
            adversely affected.
                                         15

-------
Recommendations

            We recommend that the Assistant Administrator for Administration and Resources
            Management require the Director, Office of Acquisition Management to:

             3-1.    Periodically review the accuracy of the contracting data contained within
                    ICMS, including service codes, and make necessary corrections in a timely
                    manner.

             3-2.    Periodically compare ICMS and FPDS data to ensure the reliability of the
                    information, and examine any major differences and correct in a timely
                    manner.

             3-3.    Summarize the steps taken or planned, and progress made in correcting the
                    PBSC data quality issues in the Office of Administration and Resources
                    Management s Federal Manager s Financial Integrity Act report.

Agency Comments and OIG Evaluation

            The Office of Administration and Resources Management agreed with the findings
            and recommendations and included planned corrective actions. The response
            mentions significant problems with EPA s contract tracking systems, noting that
             EPA s current systems do not meet our business needs. They provide inadequate
            reporting, cumbersome tracking status, and redundant data entries which result in
            extraordinary inefficiencies and errors.  The Office of Administration and
            Resources Management anticipates that the new system it has proposed will correct
            these problems.  The full response from the Office of Administration and
            Resources Management is  included in this report as Appendix B.
                                          16

-------
                                                                         Appendix A

               Details on Scope and Methodology
We performed this audit in accordance with the Government Auditing Standards, issued by the
Comptroller General of the United States.  We assessed EPA s policies and procedures to ensure
performance-based service contracts were awarded when possible, determined whether contract
statements of work were designed with performance-based attributes, and determined whether
PBSC information was reported accurately. We performed this audit from December 2001
through November 2002.

To assess EPA s policy and procedures, we reviewed OMB s Office of Federal Procurement
Policy guidelines and policy letters and interviewed OMB and GAO personnel concerning
performance-based contracting.  We interviewed EPA personnel at Washington, B.C.;
Cincinnati, Ohio; Research Triangle Park, North Carolina; and Philadelphia, Pennsylvania from
the following offices (the offices are in Washington unless otherwise noted):

             *  Office of Administration and Resources Management
                     "  Office of Acquisition Management
                     "  Policy, Training and Oversight Division
                     "  Headquarters Procurement Operations Division
                     "  Superfund/Resource Conservation and Recovery Act
                       Regional Procurement Operations Division
                     "  Research Triangle Park Procurement Operations Division
                     "  Cincinnati Procurement Operations Division
             „.  Office of Environmental Information
             „.  Office of Solid Waste and Emergency Response
             „.  Andrew W. Breidenbach Environmental Research Center (Cincinnati)
             „.  Region 3 (Philadelphia)

Using EPA s active contracts listings as of July 2001 (preliminary research) and June 30, 2002
(field work), we judgmentally selected a total  of 30 contracts and task orders considering contract
value, contract type, service type, and area/program/media supported, to obtain a cross-section of
EPA PBSC-eligible service contracts.  We selected 10 contracts/task orders
(approximately $16 million maximum potential value) not identified as PBSC even though
similar services had previously been identified as PBSC out of approximately 500
non-performance-based contracts ($11.7 billion maximum potential value). We also selected
20 contracts/task orders (approximately $78 million maximum potential value) out of
approximately 120 contracts ($718 million maximum potential value) that EPA identified as
PBSC.

We did not randomly select the sample because of the relatively low number of EPA
performance-based contracts (approximately 120) and the presence of many low dollar value,
similar contracts. While contract type may be a significant factor in determining whether PBSC

                                         17

-------
contracts were designed to take advantage of the benefits, we did not specifically review the
contracts to determine whether the most appropriate contract type was used in each situation.

To assess whether EPA s policies and procedures ensured performance-based service contracts
were awarded when possible, we reviewed our sample of 10 non-performance-based contracts to
determine reasons for not using the performance-based method. We also reviewed EPA s
September 4, 2002, Acquisition Forecast Database to  determine whether the performance-based
method was planned for additional  forecasted acquisitions.

To determine whether contract statements of work were designed with performance-based
attributes, we reviewed the 20 statements of work for  contracts and task orders to evaluate
whether they contained the required FAR performance-based attributes.

To determine whether EPA reported performance-based acquisitions accurately, we compared
summary-level FPDS information to EPA s PBSC actions recorded in EPA s ICMS.  For our
sample of 20 performance-based contracts, we determined whether the action was properly
recorded as PBSC in ICMS.  We also interviewed system data operators to determine reasons for
differences. However, we did not review system controls for accurate, complete, and timely data.
Prior Audit Coverage

GAO Report GAO-02-1049, Guidance Needed for Using Performance-Based Service
Contracting (September 2002), raised concerns as to whether agencies have a good
understanding of PBSC and how to take full advantage of its benefits.  GAO reported that
various agency officials noted the need for better PBSC guidance and criteria on which contracts
should be called  performance-based.  OMB s Office of Federal Procurement Policy recognized
this problem and is developing new guidance designed to improve agency use of PBSC.
                                          18

-------
                                                                        Appendix B
        ^
\
              Agency Response  to the Draft Report
                  UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
                                WASHINGTON, D.C.  20460
                                       MAR 19 2003
 MEMORANDUM
 SUBJECT:
 FROM:
 TO:
                                                                      OFFICE OF
                                                                    ADMINISTRATION
                                                                    AND RESOURCES
                                                                      MANAGEMENT
Draft Audit Report
EPA Could Increase Savings and Improve Quality
Through Greater Use of Performance-Based Service Contracts
Assignment No. 2002-000319

Morris X. Winn lotted!
Assistant Administrator

Robert Mitchell
Director for Contracts Audit
Office of Inspector General
       Thank you for the opportunity to provide comments on the subject draft audit report We
 are in general agreement with the findings, and we appreciate your support in helping us increase
 the use of performance-based service contracting (PBSC) to meet the Agency s goals.

       As you are aware, the Administration has set Government-wide goals, which cannot be
 met without Agency-wide support and participation. With this is mind, we have taken several
 initiatives to promote and market PBSC throughout the Agency. These initiatives, which
 strongly emphasize a sharing of the responsibility with our program office customers, include:
 providing additional, more focused training upon request; providing improved PBSC outreach,
 direction and tools to make PBSC more user friendly to EPA offices; providing hands-on
 assistance in converting additional program requirements to PBSC; and sharing lessons learned
 and best practices throughout the Agency.  Again, we appreciate any assistance that the OIG can
 provide in helping to obtain much-needed buy-in from senior level officials.
                                         19

-------
       Detailed comments on the report are provided below. While we generally agree with
your draft report, we take issue with several specific findings (see comments below). Our
responses are discussed by subject in the order of the appearance in the report

       If you have any questions or comments, please call Judy Davis. Director, Office of
Acquisition Management, at 564-4310 or Joan Wooley at 564-4732.
Chapter 2: EPA Needs to Increase Use of Performance-Based Service Contracting

We agree that EPA needs to increase performance-based service contracting. However, the
report states that of the $599M in PBSC-eligible obligations for the 9-month period reviewed, it
was determined that EPA could have saved as much as $72.5M by better applying PBSC. The
report does not identify how the $599M figure was determined.  It would be helpful to us if the
OIG explained how the $599M figure was calculated.

Page 4. Finding: EPA Could Award More Performance-Based Contracts

We agree with the overall finding; however, we disagree with the following and have provided
our rationale.

Contract  68D01051 for Cafeteria Services and Operation of EPA employee store. This contract
is a license agreement for the contractor to operate the cafeteria and employee store.  A small
subsidy for the operation of the cafeteria is provided by the Government. The subsidy was
designed to decrease, based on the expectation of the contractor making a profit once the EPA
facility was fully occupied. With the decreasing subsidy and the results-oriented SOW, we
believe that the agreement is more in line  with a PBSC contract. However, the nature of the
agreement does not lend itself to specific performance standards (other than those specified by
law (sanitation, business  licenses, etc.). Therefore no written incentives or disincentives were
included  (other than the incentive for the contractor to make enough money to pay for all
expenses). Based on the facts provided we recommend that this contract be deleted from
table 2-1.

Contract  68-W-00-128 for Voice Mail System Maintenance. We agree that this was not awarded
fully as a performance-based contract. However, we believe that the contract does contain many
performance-based elements. For example, the statement of work is written in terms of what
the output must be rather than how the work is to be accomplished, and measurable
performance standards are defined for all  major task areas. The contract is scheduled for
recompete in 2004 and we have started early discussions with the program office to ensure that
the follow-on contract is  awarded as full performance-based.

Contract  68-W-00-125 Technical Support for Compliance Assurance. We agree that some tasks
under this contract could be performance-based. However, at the time the contract was awarded,
based on the uncertainties associated with the level of effort and other variables, it would have
been difficult to convert the statement of work (SOW) to PBSC. The contract expires
                                          20

-------
September 2003, and we have been advised by the program office that there will be no follow-on
contract for these services.  We will however ensure that where feasible, task orders are PBSC.

Page 7. Finding: Contracts That Were Prescriptive

In our efforts to transition to PBSC some of our contracts may have been prescriptive; however,
we disagree with the following:

Contract 68-C-00-189 and Contract D-98-169 for Security Guard Services.  We disagree with the
findings and recommend that these contracts be deleted from table 2-3 as being contracts that
were prescriptive. Since the events of September 11th we had to modify our guard services
contracts to clearly specify our requirements.  We no longer consider these services to be low
risk, nor do we believe that the requirements restrict the contractor from using innovative
approaches or shifting responsibility.

Contract 68-W-02-020 Property Management Services. We disagree that this contract contained
PBSC attributes which were overly prescriptive and recommend that this contract be deleted
from the list of contracts identified as being prescriptive. The SOW included a brief background
and purpose, then provided generalized support requirements, further delineated by specific
performance objectives/appropriate performance measures/performance standards. Although we
included a description of the labor categories, we do not believe that this impedes the
contractor s ability to offer innovative approaches to determine how best to perform.

Page 7/8. Finding: Incentives and Disincentives not Meaningful.

Contract 68-W-00-104 Security Support Services. We agree with the OIG that this contract,
awarded in June 2000, contains numerous disincentives of nominal dollar value. Working with
the program office, we will review all of the incentives and disincentives in this contract and
revise them as appropriate to be more meaningful. By June 30, 2003, we will ensure that the
revised incentives and disincentives are incorporated into the contract for option period two.

Page 14 Finding: Performance-Based Task Orders Issued Under Non-Performance-Based
Contracts Not Shown as PBSC in FPDS.

We agree that some data was not properly coded as PBSC or non-PBSC. Corrections have now
been made to ICMS data, based on a review of ICMS data by OAM  s operations divisions.
Those changes have been re-submitted to FPDS and have been processed.  We have not yet seen
the results.

We also agree that PBSC task orders under non-PBSC contracts were not being submitted as
PBSC to FPDS.  This discrepancy accounted for mostof the under-reporting of PBSC actions to
FPDS. EPA has corrected its reporting process and has resubmitted the PBSC task order actions
to FPDS. Due to a six week shutdown of FPDS, the re-submission has just been processed.
Again, we have not yet seen the results.
                                          21

-------
Page 15 Finding: Reporting Improvements Needed.

The report states: Until EPA ensures that ICMS data is consistent with FPDS, the Agency s
credibility as well as any subsequent stakeholder decision making could be adversely affected.
ICMS data changes daily, while FPDS is currently updated only quarterly.  One of the E-Gov
initiatives planned for FY2004 is development of a new and better data gathering system, Federal
Procurement Data System - Next Generation (FPDS-NG). This system will be updated more
frequently - possibly overnight, but a final decision has not been made. After FPDS-NG is
implemented, it is more likely that the two systems will more closely match.

It should also be noted that EPA s  current systems do not meet our business needs. They provide
inadequate reporting, cumbersome tracking status, and redundant data entries which result in
extraordinary inefficiencies and errors. We need a system that can interface with Government-
wide systems as described in the ACQUISITION SYSTEM Project proposed in the FY 2005
OAM Capital Planning for Investment Control Process (CPIC).

The proposed system will be a secure web-based tool, eliminate duplicate data entry, track actual
costs and progress in real time, and provide a Commercial Off The Shelf (COTS) application
consistent with the President s E-Gov initiatives.  We appreciate the OIG s help in soliciting
support for this system.

RECOMMENDATIONS

The OIG recommends that the Assistant Administrator for Administration and Resources
Management, in coordination with the Senior Resource Officials for each Headquarters
and regional office and other field  locations promote maximum use of PBSC by:

Recommendation 2-1

OIG:  Developing and implementing a PBSC plan that includes short-term and long-term
strategies for maximizing the use of performance based contracting. The plan should establish
goals for PBSC use in program offices and monitor progress  against those goals.

OARM s Response: We concur with this recommendation and will provide a plan of action to
comply with the recommendation in  the final report.  It would be helpful if the IG would assist us
in soliciting support from senior officials in the Agency to engage AAs and RAs in helping the
Agency meet its goals.

Recommendation 2-2

OIG:  Developing and implementing procedures to improve the use of PBSC, such as deploying
experienced PBSC contracting personnel to assist program offices in developing performance-
based statements of work, and ensuring that the statements of work contain the required
elements.
                                          22

-------
OARM s Response:  Although we concur with this recommendation, we currently provide
experienced PBSC contracting personnel to assist program offices, develop PBSC SOW
templates, etc. We will provide details in the final report.

Recommendation 2-3

OIG: Identifying forums, such as the Innovation Action Council, to help further promote PBSC,
and to discuss PBSC best practices and issues on a regular basis.

OARM s Response:  We concur with the recommendation.  We currently engage in several
venues to promote PBSC. We solicit the IG support in getting buy-in at the senior level.

Recommendation 2-4

OIG: Developing and implementing a procurement policy addressing the FAR Subpart 7.105
requirement to implement performance-based contracting methods, or provide a rationale for not
using those methods.

OARM s Response: We concur with this recommendation.

The OIG recommend that the Assistant Administrator for Administration and Resources
Management require the Director, Office of Acquisition Management to:

Recommendation 3-1

OIG: Periodically review the accuracy of the contracting data contained within ICMS, including
service codes, and make necessary corrections in a timely manner.

OARM s Response:  We concur with this recommendation. Please see comments referenced
under the findings on page 14.

Recommendation 3-2

OIG: Periodically compare ICMS and FPDS data to ensure the reliability of the information, and
examine any major differences and correct in a timely manner.

OARM s Response: We concur with this recommendation.

Recommendation 3-3

OIG: Summarize the steps taken or planned, and progress made in correcting the PBSC data
quality issues in OARM s Federal Manager s Financial Integrity Act report.

OARM s Response:  We concur with this recommendation  and will address this and other data
quality issues in the Federal Manager s Financial Integrity Act report.
                                          23

-------
24

-------
                                                                        Appendix C
                            Report Distribution
Assistant Administrator for Administration and Resources Management
Assistant Administrator for Environmental Information
Assistant Administrator for Solid Waste and Emergency Response
Assistant Administrator for Air and Radiation
Assistant Administrator for Water
Assistant Administrator for Prevention Pesticides and Toxic Substances,
Assistant Administrator for International Affairs
Assistant Administrator for Research and Development
Agency Followup Official
Agency Audit Followup Coordinator
Associate Administrator for Congressional and Intergovernmental Relations
Associate Administrator for Public Affairs
Director, Office of Acquisition Management
Director, Office of Emergency and Remedial Response
Audit Liaison, Office of Administration and Resources Management
Audit Liaison, Office of Environmental Information
Audit Liaison, Office of Solid Waste and Emergency Response
Inspector General (2410)
Regional Administrators, Regions 1 through 10
Regional Audit Followup Coordinators, Regions 1 through 10
                                         25

-------