\saA OIG % PPO^ OFFICE OF INSPECTOR GENERAL Catalyst for Improving the Environment Audit Report EPA Could Increase Savings and Improve Quality Through Greater Use of Performance-Based Service Contracts Report No. 2003-P-00008 March 31, 2003 ------- Report Contributors: Herbert Maletz Denton Stafford Charles Watts Brian Lick Abbreviations EPA Environmental Protection Agency FAR Federal Acquisition Regulation FPDS Federal Procurement Data System GAO General Accounting Office ICMS Integrated Contracts Management System OIG Office of Inspector General OMB Office of Management and Budget PBSC Performance-Based Service Contracting START Superfund Technical Assessment and Response Team ------- v. UNITED STATES ENVIRONMENTAL PROTECTION AGENCY '. WASHINGTON, DC 20460 '"( OFFICE OF INSPECTOR GENERAL March 31,2003 MEMORANDUM SUBJECT: Audit Report: EPA Could Increase Savings and Improve Quality Through Greater Use of Performance-Based Service Contracts Report No. 2003-P-00008 FROM: Robert Mitchell Director for Contract Audits TO: Morris X. Winn Assistant Administrator Office of Administration and Resources Management This is our final report on the subject audit conducted by the Office of Inspector General (OIG) of the U.S. Environmental Protection Agency (EPA). This audit report contains findings that describe the problems the OIG has identified and corrective actions the OIG recommends. This audit report represents the opinion of the OIG and the findings contained in this report do not necessarily represent the final EPA position. Final determinations on matters in this audit report will be made by EPA managers in accordance with established audit resolution procedures. Action Required In accordance with EPA Order 2750, you are required to provide a written response to the findings and recommendations presented in this draft report within 90 calendar days of the date of this report. You should include a corrective actions plan for agreed-upon actions, including milestone dates. We have no objection to the further release of this report to the public. For your convenience, this report will be available at http://www.epa.gov/oig/eroom.htm. If you or your staff have any questions regarding this report, please contact me at (202) 566-0891 or Herbert Maletz at (212) 637-3058. ------- ------- Executive Summary Purpose Performance-Based Service Contracting (PBSC)is an acquisition methodology in which the Government pays for results rather than effort or process. Office of Management and Budget (OMB) studies have shown that PBSC saves money and improves contractor performance. While PBSC has existed for many years, it has not been widely adopted throughout the Federal Government. We conducted this audit to determine whether the Environmental Protection Agency (EPA) effectively implemented PBSC to the maximum extent possible. Our specific objectives were to determine whether EPA was: " Awarding performance-based service contracts whenever possible and designing the contracts to take full advantage of the benefits. " Reporting accurate PBSC information to OMB. Results of Review EPA has made limited use of PBSC. Generally, EPA has only used PBSC to obtain commercially available services, such as janitorial and landscaping work. We found that EPA awarded contracts that were not performance-based for services previously awarded as performance-based. Also, many contracts EPA identified as performance-based were not designed to take advantage of PBSC s benefits. These contracts were too prescriptive in how the work was to be carried out or did not provide meaningful incentives or disincentives. Based on discussions with Agency officials, this occurred because EPA did not adequately plan or hold officials accountable for increasing PBSC use, and there was a general reluctance to shift the responsibility for outcomes to contractors. As a result, EPA missed opportunities to achieve greater cost savings and improve contractor performance. Of the $599 million in PBSC-eligible obligations for the 9-month period reviewed, we determined that EPA could have saved as much as $72.5 million by better applying PBSC. Also, OMB s Federal Procurement Data System (FPDS) showed nearly 50 percent less PBSC obligations than EPA reported in its own Integrated Contracts Management System (ICMS). This primarily occurred because FPDS did not show performance-based task orders issued under non-performance-based contracts as PBSC; EPA made the necessary corrections in ICMS but had not submitted them to FPDS. In addition, EPA officials said ICMS lacked the capability to provide all data needed in FPDS. As a result, EPA s contracting ------- actions were inaccurately portrayed in the national database, which serves as a basis for reporting to Congress, the President, and the general public; and for performing analysis. Recommendations We recommend that the Assistant Administrator for Administration and Resources Management take certain actions to improve accountability and procedures in relation to PBSC. We also recommend that the Assistant Administrator take actions so that the accuracy of ICMS is reviewed and that ICMS and FPDS data is periodically compared. Agency Comments and OIG Evaluation On February 14, 2003, we issued a draft report to the Office of Administration and Resources Management, and received a response on March 19, 2003 (see Appendix B). The Office generally agreed with the findings and recommendations and included planned corrective actions. The Office acknowledged significant problems with contract tracking systems, and noted a new system it has proposed will correct the problems. In its response, the Office also provided additional information, and we clarified our final report as needed. ------- Table of Contents Executive Summary Chapters 1 Introduction 1 2 EPA Needs to Increase Use of PBSC 3 3 EPA Needs to Improve PBSC Reporting 13 Appendices A Details on Scope and Methodology 17 B Agency Response to the Draft Report 19 C Report Distribution 25 ------- ------- Chapter 1 Introduction Purpose Performance-Based Service Contracting (PBSC) is an acquisition methodology in which the Government pays for results rather than effort or process. Office of Management and Budget (OMB) studies have shown that PBSC saves money and improves contractor performance. While PBSC has existed for many years, it has not been widely adopted throughout the Federal Government. We conducted this audit to determine whether the Environmental Protection Agency (EPA) effectively implemented PBSC to the maximum extent possible. Our specific objectives were to determine whether EPA was: " Awarding performance-based service contracts whenever possible and designing the contracts to take full advantage of the benefits. " Reporting accurate PBSC information to OMB. Background PBSC allows contractors to propose the most innovative, effective, efficient, and economical approach their business experience and ability will allow. Such contracts can help an agency avoid cost overruns, schedule delays, and failure to achieve specified results. EPA participated in OMB s PBSC studies as well as its own pilot projects and has experienced these benefits. OMB s Office of Federal Procurement Policy has encouraged greater use of PBSC for over 10 years. In April 1991, this Office issued Policy Letter 91-2, Service Contracting. This Policy Letter defined PBSC, required the use of PBSC where practicable, and required agencies to better match their acquisition and contract administration strategies to the specific requirements. Policy Letter 91-2 was incorporated into the Federal Acquisition Regulation (FAR) in 1997. Subpart 37.1 of the FAR, Service Contracts-General, states when acquiring services, agencies must use performance-based contracting methods to the maximum extent practicable, except for architect-engineer services, construction, utility services, and services incidental to supply purchases. ------- FAR Subpart 37.6, Performance-Based Contracting, first issued in 1997, prescribes policies and procedures for PBSC. The FAR notes performance-based contracts: " Describe the requirements in terms of results required rather than the methods of performing the work; " Use measurable performance standards (i.e., terms of quality, timeliness, quantity, etc.) and quality assurance surveillance plans; " Specify procedures for reductions of fee or for reductions to the price of a fixed-price contract when services are not performed or do not meet contract requirements; and " Include performance incentives where appropriate. OMB s Office of Federal Procurement Policy issued Information on Best Practices for Performance-Based Service Contracting in October 1998. This guide notes studies have shown that PBSC has the potential to improve customer satisfaction by more than 18 percent and achieve average contract cost savings of 15 percent. However, the Office noted that application of only selected aspects of the PBSC methodology can limit benefits. Agencies must report PBSC information to the Federal Procurement Data System, which is the central repository of statistical information on Federal contracting. The system contains detailed information on contract actions over $25,000, and summary data on procurements less than $25,000. Scope and Methodology From October 2001 through June 2002, EPA acquired approximately $599 million in services that were PBSC-eligible. This information was obtained from the November 7, 2002, Integrated Contracts Management System (ICMS). We performed this audit in accordance with Government Auditing Standards, issued by the Comptroller General of the United States. We judgmentally selected a sample of 30 contracts and task orders for review. We assessed EPA s policies and procedures to ensure performance-based contracts were awarded when possible, determined whether contracts statements of work were designed with performance-based attributes, and determined whether PBSC information was reported accurately. During the audit, we interviewed numerous personnel. We performed this audit from December 2001 through November 2002. As part of our review, we took into account a prior General Accounting Office (GAO) report (GAO-02-1049, Guidance Needed for Using Performance-Based Service Contracting, September 2002). Details on our scope and methodology are in Appendix A. ------- Chapter 2 EPA Needs to Increase Use of PBSC EPA has made limited use of PBSC. Generally, EPA has used PBSC to obtain commercially available services, such as janitorial and landscaping work. We found that EPA awarded non-performance-based contracts for services previously awarded as performance-based. Also, many contracts EPA identified as performance-based were not designed to take advantage of PBSC s benefits. These contracts were too prescriptive in how the work was to be carried out or did not provide meaningful incentives or disincentives. Based on discussions with Agency officials, these conditions occurred because EPA did not adequately plan or hold officials accountable for increasing PBSC use, and there was a general reluctance to shift the responsibility for outcomes to contractors. As a result, EPA missed opportunities to achieve greater cost savings and improve contractor performance. Of the $599 million in PBSC-eligible obligations for the 9-month period ended June 30, 2002, we determined that EPA could have saved as much as $72.5 million by better applying PBSC. OMB Has Encouraged Greater Use of PBSC Since 1991 As noted in Chapter 1, OMB has required the use of PBSC where practicable since 1991. In 1997, this requirement was incorporated into the FAR, Subpart 37.6, Performance-Based Contracting. PBSC is intended to ensure that required performance quality levels are achieved and that total payment is related to the degree that services performed meet contract standards. Performance-based contracts describe the desired end result (outcomes) expected of the contractor, but do not direct how the work is to be performed. In April 2002, FAR Subpart 7.105, Contents of Written Acquisition Plans, required that acquisition planners develop written acquisition plans for service contracts or task orders that describe strategies for implementing performance- based contracting methods, or provide rationale for not using those methods. EPA s Contracts Management Manual, Chapter 1 (February 1999) notes EPA s Senior Resource Officials are the primary acquisition planners. EPA Has Made Limited Use of PBSC Generally, EPA has limited PBSC to services performed widely in the commercial sector: custodial services, building maintenance, or landscaping. As noted in a GAO report, this was the case with many agencies. These services easily lend themselves to PBSC because measuring and specifying outcomes are relatively straightforward. Additionally, these generally non-complex services do not present substantial risk to the Government. ------- OMB has set Government-wide goals for PBSC. Beginning in fiscal 2001, 10 percent of all eligible service contracting obligations were to be performance- based. The PBSC goals incrementally increase 10 percent annually until 2005. The goal for fiscal 2002 was 20 percent. For fiscal 2001, EPA reported approximately 11 percent of its eligible contract obligations were performance-based, and thus met its goal of 10 percent. However, for fiscal 2002, EPA only reported that 16.6 percent of contract obligations were performance-based, short of OMB s goal of 20 percent. EPA needs to increase its effort to meet OMB s PBSC goals. EPA Could Award More Performance-Based Contracts Between 1998 and 2001, EPA awarded many contracts without employing PBSC attributes, for services similar to previously awarded EPA performance-based contracts. We reviewed 10 such contracts (with a maximum potential value of $16.3 million), as shown in Table 2-1: Table 2-1: Contracts Not Performance-Based for Which Similar Services Were Previously Awarded as Performance-Based Contract Maximum Number Service Potential Value 68-D-01-051 68-W-00-128 68-W-00-125 68-W-99-078 68-D-99-026 68-D-99-014 68-W-99-012 68-D-98-166 68-D-98-092 68-D-98-029 Total Cafeteria Services and Operation of EPA Employee Store Voice Mail System Maintenance Technical Supportfor Compliance Assurance Facility Operatbn and Maintenance Service for trie Manchester EPA Laboratory Landscape/Grounds Maintenance Support National Performance Audit Program (Air Monitoring) Perimeter Guard Services Security Guard Services Peer Review, Workshops, Training Sessions, and Conference Support Maintenance Services for the Natbnal Airand Radiation Laboratory $264,000 677,000 2,813,000 916,000 162,000 7,127,000 3,218,000 505,000 292,000 284,000 $16,258,000 Most of these contracts were for services widely performed in the commercial sector, and did not present substantial risk to the Agency. Measuring and specifying outcomes for these types of services were relatively straightforward. Based on the types of services acquired and the fact that EPA had previously awarded a performance-based contract of a similar nature, we believe these contracts could have been performance-based. ------- EPA Office of Acquisition Management officials said the above contracts were not performance-based for a variety of reasons, including urgency of need, inability to define performance objectives, and inexperience in using PBSC methods (some earlier contracts). The officials also stated the Voice Mail System Maintenance contract was not fully performance-based primarily because staff did not have enough time to convert the statement of work. In addition to the 10 non-performance-based contract files reviewed, we also selectively reviewed ICMS data as of June 30, 2002, and found instances where EPA awarded non-performance-based contracts even though contracts for similar services were previously performance-based: " ICMS reported four non-performance-based facilities operation support services contracts, even though at least seven other facilities operation support services contracts within the past 5 years were performance-based. " Two landscaping contracts were not identified as performance-based, even though in the last 2 years EPA awarded at least two other landscaping contracts identified as performance-based. " EPA awarded a janitorial/custodial contract not identified as performance based, even though within the last 5 years EPA awarded at least nine other performance-based janitorial/custodial contracts. " EPA awarded a non-performance-based mail service contract, even though within the last 5 years EPA has awarded at least five other performance-based mail services contracts. Performance-Based Contracts Not Always Designed to Take Advantage of Benefits Performance-based contracts were not always designed to take advantage of PBSC benefits. We reviewed 20 individual performance-based contracts and task orders ($77.8 million maximum potential value), which generally contained at least one of the following performance-based attributes required by FAR, Subpart 37.6: " Performance requirements described in terms of results required. " Measurable performance standards and quality assurance surveillance plans. " Financial disincentives. " Performance incentives. Eight of the contract actions appeared to generally be designed to take advantage of PBSC benefits, even though they may not have contained all four of the above ------- PBSC attributes. However, in our opinion, the other 12 contracts were not designed to take advantage of PBSC s benefits. These contracts PBSC attributes were either prescriptive in how the work was to be carried out or did not provide meaningful incentives to increase contractor efficiency or penalize contractors for poor performance. Performance-Based Contracts Designed to Take Advantage of PBSC Benefits The following eight performance-based contracts appeared to generally be designed to take advantage of PBSC benefits (see Table 2-2). These contracts exhibited many of the basic PBSC attributes. Table 2-2: Performance-Based Contracts Designed to Take Advantage of PBSC Benefits Contract Number 68-C-01-025 68-D-01-049 68-S3-00-02 Work Assmnt. # SE3-02-03-010 68-W-00-075 68-W-00-095 68-W-01-018 Task Order # 005 68-W-01-059 68-W-02-020 Total Maximum Service Potential Value Study to Determine Concentrations of Pollutants in Waste Water Janitorial Services for Facilities START (Superfund Technical Assessment and Response Team) Removal Site Evaluation Contract Laboratory Program-Analytical Support Energy Star Hotline and Distribution Center ESAT (Environmental Services Assistance Team) Performance Evaluation and Field Support Services Warehousing and Storage Services Property Management Services $48,500 732,000 39,000 2,245,000 2,982,000 8,148,000 13,557,000 2,117,000 $29,868,500 The START Removal Site Evaluation contract action illustrates how PBSC can provide benefits. EPA only specified the desired end result, not the process to be used. EPA obtained two competing proposals and found both to be technically sound. However, while one firm proposed using standard sampling technology, the other firm proposed using a new technology that had proven successful at another site. The proposals differed significantly in estimated costs: the estimate for the standard sampling technology was $154,000 while the new technology estimate was $39,000, or $115,000 less. EPA selected the lower cost proposal. By not being prescriptive, EPA opened the door to more innovative approaches that cost less and possibly improved service. ------- While the eight contract actions in Table 2-2 represent EPA s best PBSC efforts, it is not our intent to suggest that the attributes could not be improved. Contracts That Were Prescriptive Seven contracts/actions, although identified as performance-based, were prescriptive in how the work was to be performed, and frequently required direction from EPA project officers (see Table 2-3). Most of these contracts were for traditional, relatively low-risk services. Table 2-3: Contracts That Were Prescriptive Contract Maximum Number Service Potential Value 68-C-00-137 68-C-00-189 68-D-98-169 68-D-00-200 68-S3-99-04 Task Order # 12 68-W-99-082 68-W-01-002 Total Study to Estimate Selected Chemical Levels in Fish Tissue Security Guard Services Security Guard Services Facility Maintenance and Equipment Operation of Lab ERRS (Emergency Rapid Response Service) Contamination Removal Administrative Support Services Information Management Support Services $13,800,000 3,000,000 774,000 4,600,000 1,900,000 2,300,000 343,000 $26,717,000 On the surface, these contracts appeared to contain the PBSC attributes. However, a closer review indicated prescriptive statements of work that might have impeded the contractor s innovation and freedom to decide how best to perform the work. For example, one guard services contract (68-C-00-189) prescribed the number of guards desired, specific posts to be manned, specific sizes for uniform patches and vehicle lettering, and specific age of all guard vehicles. These examples illustrate EPA s reluctance to allow contractors the flexibility to use innovative approaches or shift responsibility for outcomes. Incentives and Disincentives Not Meaningful Five contracts/actions, which EPA identified as performance-based, did not have meaningful incentives and/or disincentives. Although two provided negative incentives, the proposed penalties were low dollar amounts. The other three did not include any incentives (see Table 2-4). ------- Table 2-4: Incentives/Lack of Incentives Not Meaningful Contract Maximum Number Service Potential Value 68-C-99-252 68-C-99-254 68-W-00-104 68-W7-0055 Task Order #408 68-W7-0055 Task Order #422 Total Environmental Impact Analysis Administrative Support Services Security Support Services Library Systems Support Operations of National Computer Center $1,996,000 2,573,000 11,413,000 101,000 5,106,000 $21,189,000 For example, while the administrative support services contract (68-C-99-254) exhibited PBSC attributes, its negative incentives were not meaningful. The tasks were organized in such a way that minuscule penalties could be assessed. Specifically, the penalty assessed against poor performance for inventorying the stockroom was 0.4 percent of $4,004.33 per month, or $16.02. Superfund PBSC While most of EPA s current performance-based contracts are for less complex and lower risk services that are widely available in the commercial sector, its largest contracts have been more complex, higher risk, cost reimbursable, level of effort types, especially for the Superfund Program. These contracts place more of the risk for cost and performance on the Government; contractors are only required to give their best effort, and all reasonable and allowable costs are reimbursed. The Office of Acquisition Management and Office of Emergency and Remedial Response developed PBSC model statements of work templates for various Emergency Rapid Response Service, START, and Response Action Contract assignments and tasks that take place on a regular basis. According to Office of Emergency and Remedial Response officials, the overall structure of these multi-year time and materials contracts was not developed using the PBSC methodology, and the use of these templates was optional. These templates were developed in 2002 as a temporary measure to increase PBSC use under the current contracts, and were not widely used at the time of our review. Further, these templates by design did not significantly change current contract management practices of relying upon close EPA contractor oversight. Therefore, until the basic contract incorporates all PBSC features and is managed accordingly, EPA may not maximize PBSC benefits. ------- More PBSC Planning and Accountability Needed To promote the use of PBSC, EPA s Office of Acquisition Management relied on a marketing strategy involving training and assistance to program offices. However, EPA did not adequately plan or hold responsible officials accountable for increasing PBSC s use. We believe this is primarily due to Agency officials reluctance to shift the responsibility for outcomes to contractors. Current PBSC Planning Not Specific Enough To meet the PBSC goals, the Office of Acquisition Management developed a short-term plan for fiscal 2002, which focused on: " Sharing responsibility for meeting the goal with program office customers; " Implementing additional, more focused PBSC training; " Providing improved PBSC outreach, direction, and tools to make PBSC more user friendly to EPA offices; " Converting additional program requirements to PBSC; and, ultimately, " Sharing lessons learned and best practices throughout the Agency. This plan generally emphasized greater PBSC use throughout EPA, but there was no high-level accountability to ensure the plan s implementation. The plan did not outline a detailed strategy to identify additional program requirements that could be converted to PBSC, although such a strategy is planned for 2003. Furthermore, since EPA contracts generally last 3 years or longer, EPA s plan should also include a long-term PBSC strategy to address contracts that will not need to be replaced for some time. EPA s Acquisition Forecast Database could help identify planned performance- based contracts. This web site provides the business community with information about upcoming EPA procurements. As of September 4, 2002, of the 204 planned contracts listed, 38 (19 percent) were identified as performance-based. Our review of the list showed that some traditional services, such as library operations, were not identified as performance-based. Lack of Accountability For PBSC Within EPA EPA s Office of Acquisition Management has attempted to market and promote PBSC, but has experienced difficulties with obtaining firm program office commitments on future performance-based acquisitions. The Office s staff said this is due to the general reluctance of program officials to change their current culture of managing large, cost-reimbursable, level-of-effort-type contracts. Program officials echoed this sentiment, expressing concerns about major changes and the belief that they would be losing control of their business. While the Office of Acquisition Management is primarily responsible for all EPA ------- contracting activity, the Agency has not held program officials accountable to ensure the adoption and implementation of PBSC agency-wide. For example, another agency (the U.S. State Department) issued a directive that all new service contracts must be performance-based, with any deviations justified in writing and approved. EPA may need such a commitment from their Senior Resource Officials, the primary acquisition planners. EPA Pilot Project Exhibited Benefits EPA was part of OMB s PBSC pilot project from 1994 to 1998 to encourage Government-wide acceptance of PBSC and to evaluate its effects. EPA participated with two contracts one for operation of a telephone hotline and the other for training. The contract price was reported to decrease by 31 percent for the hotline and 6 percent for training. In Fiscal 1997, with assistance from OMB, EPA conducted a PBSC study within the Superfund program. Nine projects were involved in this study, and most appeared to save money. For example, at the Oronogo-Duenweg Mining Site, PBSC was responsible for lowering the project costs by 30-35 percent while improving the quality of work. Significant Savings Could Be Achieved During our review, we found that EPA missed opportunities to achieve greater cost savings and higher quality service. For the 9-month period reviewed (October 1, 2001, through June 30, 2002), EPA reported in ICMS (as of November 7, 2002) that it obligated a total of $599 million for eligible performance-based services, with $116 million labeled performance-based. OMB studies have shown that PBSC has the potential to improve customer satisfaction by more than 18 percent, and achieve average contract cost savings of 15 percent. Applying this potential savings to EPA s remaining obligations eligible to be performance-based ($483 million) indicates that EPA could have saved as much as $72.5 million during this 9-month period while improving the quality of services obtained. Recommendations We recommend that the Assistant Administrator for Administration and Resources Management, in coordination with the Senior Resource Officials for each Headquarters and regional office and other field locations, promote maximum use of PBSC by: 2-1. Developing and implementing a PBSC plan that includes short-term and long-term strategies for maximizing the use of performance based contracting. The plan should establish goals for PBSC use in program offices and monitor progress against those goals. 10 ------- 2-2. Developing and implementing procedures to improve the use of PBSC, such as deploying experienced PBSC contracting personnel to assist program offices in developing performance-based statements of work, and ensuring that the statements of work contain the required elements. 2-3. Identifying forums, such as the Innovation Action Council, to help further promote PBSC, and to discuss PBSC best practices and issues on a regular basis. 2-4. Developing and implementing a procurement policy addressing the FAR Subpart 7.105 requirement to implement performance-based contracting methods, or provide a rationale for not using those methods. Agency Comments and OIG Evaluation The Office of Administration and Resources Management agreed with the findings and recommendations and included planned corrective actions. The Office also provided additional information, and we clarified the report as needed. The full response is included in this report as Appendix B. 11 ------- 12 ------- Chapter 3 EPA Needs to Improve PBSC Reporting OMB s Federal Procurement Data System (FPDS) showed nearly 50 percent less PBSC obligations than EPA reported in its own ICMS. This primarily occurred because FPDS did not show performance-based task orders issued under non- performance-based contracts as being performance-based. EPA made the necessary corrections in ICMS but had not submitted them to FPDS. In addition, EPA officials said ICMS lacked the capability to provide all data needed in FPDS. As a result, EPA s contracting actions were inaccurately portrayed in the national database, which serves as a basis for reporting to Congress, the President, and the general public, and for performing analysis. Requirements Public Law 93-400, Office of Federal Procurement Policy Act, requires the Office of Federal Procurement Policy to establish a system for collecting, evaluating, and disseminating Federal contract information. FPDS is that central repository. FAR Subpart 4.6, Contract Reporting, requires each executive agency to establish and maintain for a period of 5 years a computer file, by fiscal year, containing unclassified records of all procurements exceeding $25,000. Agencies must transmit this information to FPDS. The FPDS Reporting Manual outlines PBSC information reporting requirements, and states that PBSC applies only to specific service codes. Any procurement action coded as a product or as any of the following services is not eligible to be categorized as PBSC in FPDS: " Architect-engineer services. " Construction. " Utility services. " Services that are incidental to supply purchases. FAR Subpart 37.102 states agencies are not required to use performance-based contracting methods for these services. For PBSC reporting purposes, these services are considered not PBSC eligible. ICMS is EPA s automated acquisition system and contains information on EPA s contracting activities. EPA converts ICMS information to a format compatible with FPDS, and this information is then sent to FPDS quarterly. To report accurate PBSC information, the Agency must determine whether the procurement action 13 ------- meets PBSC eligibility requirements, contains required attributes, and uses an appropriate service code. EPA Under-Reported PBSC Activity Our comparison of ICMS and FPDS summary level data for October 1, 2001, to June 30, 2002, disclosed that almost 50 percent of EPA s PBSC information was not shown as such in FPDS (see Table 3-1). EPA s total obligations in the two systems, as well as its PBSC-eligible obligations, were within 10 percent of each other. Table 3-1: Comparison of ICMS and FPDS Data ($ in millions) EPA Obligations Total Obligations Not PBSC Eligible PBSC Eligible PBSC ICMS $649 50 599 $116 FPDS $594 45 549 $61 Percent Under-Reported 8.5% 10.0% 8.0% 47.5% The PBSC category are obligations for procurement actions that EPA has determined met PBSC requirements. Performance-Based Task Orders Issued Under Non-Performance- Based Contracts Not Shown as PBSC in FPDS FPDS reported $55 million less PBSC obligations than ICMS. ICMS shows performance-based task orders issued under non-performance-based contracts, but these actions were not shown in FPDS as PBSC. ICMS reported approximately $43 million in performance-based task order obligations issued under non- performance-based contracts that were not shown as PBSC in FPDS. For example, ICMS reported that a national telecommunication and computer services contract (68-W7-0055) contained approximately $29 million of performance-based task orders, but that amount was not shown as such in FPDS. EPA Office of Acquisition Management officials acknowledged ICMS and FPDS inconsistencies in reporting PBSC information, and stated that ICMS information as of June 30, 2002, which was sent to FPDS, contained these errors. System users subsequently corrected the ICMS information, but EPA had not submitted these corrections to FPDS at the time of our review. These EPA officials said system users data entry is normally not reviewed for accuracy. The Office of Acquisition Management s fiscal 2002 Assurance Letter under the Federal Manager s Financial Integrity Act indicated that data quality is a major concern in EPA, and they are currently investigating these problems and are committed to their resolution. 14 ------- Other PBSC Reporting Inconsistencies Another FPDS database reporting inconsistency concerns work assignments issued under EPA contracts. EPA designed work assignments to control work flow but, unlike task orders, work assignments are not separate funding instruments from the basic contract. Therefore, FPDS does not recognize work assignments. Consequently, EPA performance-based obligations organized under work assignments in ICMS were not reported in FPDS as a performance-based obligation. Further, work assignments issued under architectural and engineering services contracts are not eligible in FPDS to be performance-based due to the service code of the basic contract. The architectural and engineering Response Action Contract for EPA Region 5 included two performance-based work assignments with a combined value of $1.3 million. While these obligations were shown in ICMS as performance-based, they were not reported in FPDS as being performance-based. The Office of Acquisition Management s fiscal 2002 Assurance Letter notes ICMS cannot be used to accurately identify and report dollars on partially performance- based work assignment contracts (where some work assignments are performance- based and some are not). During our sample selection process, we found three contracts that were incorrectly coded as performance-based due to data entry mistakes by contracting officials. While we did not verify whether there were additional coding errors, EPA officials said information entered into ICMS is not validated. EPA should ensure that PBSC coding in ICMS is as accurate as possible. Reporting Improvements Needed In accordance with FAR 4.602, FPDS is designed to serve as: " A basis for recurring and special reports to the President, the Congress, GAO, Federal executive agencies, and the general public. " A means of measuring and assessing the impact of Federal contracting on the Nation s economy. " Data for other policy and management control purposes. Until EPA ensures that ICMS data is consistent with FPDS, the Agency s credibility as well as any subsequent stakeholder decision making could be adversely affected. 15 ------- Recommendations We recommend that the Assistant Administrator for Administration and Resources Management require the Director, Office of Acquisition Management to: 3-1. Periodically review the accuracy of the contracting data contained within ICMS, including service codes, and make necessary corrections in a timely manner. 3-2. Periodically compare ICMS and FPDS data to ensure the reliability of the information, and examine any major differences and correct in a timely manner. 3-3. Summarize the steps taken or planned, and progress made in correcting the PBSC data quality issues in the Office of Administration and Resources Management s Federal Manager s Financial Integrity Act report. Agency Comments and OIG Evaluation The Office of Administration and Resources Management agreed with the findings and recommendations and included planned corrective actions. The response mentions significant problems with EPA s contract tracking systems, noting that EPA s current systems do not meet our business needs. They provide inadequate reporting, cumbersome tracking status, and redundant data entries which result in extraordinary inefficiencies and errors. The Office of Administration and Resources Management anticipates that the new system it has proposed will correct these problems. The full response from the Office of Administration and Resources Management is included in this report as Appendix B. 16 ------- Appendix A Details on Scope and Methodology We performed this audit in accordance with the Government Auditing Standards, issued by the Comptroller General of the United States. We assessed EPA s policies and procedures to ensure performance-based service contracts were awarded when possible, determined whether contract statements of work were designed with performance-based attributes, and determined whether PBSC information was reported accurately. We performed this audit from December 2001 through November 2002. To assess EPA s policy and procedures, we reviewed OMB s Office of Federal Procurement Policy guidelines and policy letters and interviewed OMB and GAO personnel concerning performance-based contracting. We interviewed EPA personnel at Washington, B.C.; Cincinnati, Ohio; Research Triangle Park, North Carolina; and Philadelphia, Pennsylvania from the following offices (the offices are in Washington unless otherwise noted): * Office of Administration and Resources Management " Office of Acquisition Management " Policy, Training and Oversight Division " Headquarters Procurement Operations Division " Superfund/Resource Conservation and Recovery Act Regional Procurement Operations Division " Research Triangle Park Procurement Operations Division " Cincinnati Procurement Operations Division „. Office of Environmental Information „. Office of Solid Waste and Emergency Response „. Andrew W. Breidenbach Environmental Research Center (Cincinnati) „. Region 3 (Philadelphia) Using EPA s active contracts listings as of July 2001 (preliminary research) and June 30, 2002 (field work), we judgmentally selected a total of 30 contracts and task orders considering contract value, contract type, service type, and area/program/media supported, to obtain a cross-section of EPA PBSC-eligible service contracts. We selected 10 contracts/task orders (approximately $16 million maximum potential value) not identified as PBSC even though similar services had previously been identified as PBSC out of approximately 500 non-performance-based contracts ($11.7 billion maximum potential value). We also selected 20 contracts/task orders (approximately $78 million maximum potential value) out of approximately 120 contracts ($718 million maximum potential value) that EPA identified as PBSC. We did not randomly select the sample because of the relatively low number of EPA performance-based contracts (approximately 120) and the presence of many low dollar value, similar contracts. While contract type may be a significant factor in determining whether PBSC 17 ------- contracts were designed to take advantage of the benefits, we did not specifically review the contracts to determine whether the most appropriate contract type was used in each situation. To assess whether EPA s policies and procedures ensured performance-based service contracts were awarded when possible, we reviewed our sample of 10 non-performance-based contracts to determine reasons for not using the performance-based method. We also reviewed EPA s September 4, 2002, Acquisition Forecast Database to determine whether the performance-based method was planned for additional forecasted acquisitions. To determine whether contract statements of work were designed with performance-based attributes, we reviewed the 20 statements of work for contracts and task orders to evaluate whether they contained the required FAR performance-based attributes. To determine whether EPA reported performance-based acquisitions accurately, we compared summary-level FPDS information to EPA s PBSC actions recorded in EPA s ICMS. For our sample of 20 performance-based contracts, we determined whether the action was properly recorded as PBSC in ICMS. We also interviewed system data operators to determine reasons for differences. However, we did not review system controls for accurate, complete, and timely data. Prior Audit Coverage GAO Report GAO-02-1049, Guidance Needed for Using Performance-Based Service Contracting (September 2002), raised concerns as to whether agencies have a good understanding of PBSC and how to take full advantage of its benefits. GAO reported that various agency officials noted the need for better PBSC guidance and criteria on which contracts should be called performance-based. OMB s Office of Federal Procurement Policy recognized this problem and is developing new guidance designed to improve agency use of PBSC. 18 ------- Appendix B ^ \ Agency Response to the Draft Report UNITED STATES ENVIRONMENTAL PROTECTION AGENCY WASHINGTON, D.C. 20460 MAR 19 2003 MEMORANDUM SUBJECT: FROM: TO: OFFICE OF ADMINISTRATION AND RESOURCES MANAGEMENT Draft Audit Report EPA Could Increase Savings and Improve Quality Through Greater Use of Performance-Based Service Contracts Assignment No. 2002-000319 Morris X. Winn lotted! Assistant Administrator Robert Mitchell Director for Contracts Audit Office of Inspector General Thank you for the opportunity to provide comments on the subject draft audit report We are in general agreement with the findings, and we appreciate your support in helping us increase the use of performance-based service contracting (PBSC) to meet the Agency s goals. As you are aware, the Administration has set Government-wide goals, which cannot be met without Agency-wide support and participation. With this is mind, we have taken several initiatives to promote and market PBSC throughout the Agency. These initiatives, which strongly emphasize a sharing of the responsibility with our program office customers, include: providing additional, more focused training upon request; providing improved PBSC outreach, direction and tools to make PBSC more user friendly to EPA offices; providing hands-on assistance in converting additional program requirements to PBSC; and sharing lessons learned and best practices throughout the Agency. Again, we appreciate any assistance that the OIG can provide in helping to obtain much-needed buy-in from senior level officials. 19 ------- Detailed comments on the report are provided below. While we generally agree with your draft report, we take issue with several specific findings (see comments below). Our responses are discussed by subject in the order of the appearance in the report If you have any questions or comments, please call Judy Davis. Director, Office of Acquisition Management, at 564-4310 or Joan Wooley at 564-4732. Chapter 2: EPA Needs to Increase Use of Performance-Based Service Contracting We agree that EPA needs to increase performance-based service contracting. However, the report states that of the $599M in PBSC-eligible obligations for the 9-month period reviewed, it was determined that EPA could have saved as much as $72.5M by better applying PBSC. The report does not identify how the $599M figure was determined. It would be helpful to us if the OIG explained how the $599M figure was calculated. Page 4. Finding: EPA Could Award More Performance-Based Contracts We agree with the overall finding; however, we disagree with the following and have provided our rationale. Contract 68D01051 for Cafeteria Services and Operation of EPA employee store. This contract is a license agreement for the contractor to operate the cafeteria and employee store. A small subsidy for the operation of the cafeteria is provided by the Government. The subsidy was designed to decrease, based on the expectation of the contractor making a profit once the EPA facility was fully occupied. With the decreasing subsidy and the results-oriented SOW, we believe that the agreement is more in line with a PBSC contract. However, the nature of the agreement does not lend itself to specific performance standards (other than those specified by law (sanitation, business licenses, etc.). Therefore no written incentives or disincentives were included (other than the incentive for the contractor to make enough money to pay for all expenses). Based on the facts provided we recommend that this contract be deleted from table 2-1. Contract 68-W-00-128 for Voice Mail System Maintenance. We agree that this was not awarded fully as a performance-based contract. However, we believe that the contract does contain many performance-based elements. For example, the statement of work is written in terms of what the output must be rather than how the work is to be accomplished, and measurable performance standards are defined for all major task areas. The contract is scheduled for recompete in 2004 and we have started early discussions with the program office to ensure that the follow-on contract is awarded as full performance-based. Contract 68-W-00-125 Technical Support for Compliance Assurance. We agree that some tasks under this contract could be performance-based. However, at the time the contract was awarded, based on the uncertainties associated with the level of effort and other variables, it would have been difficult to convert the statement of work (SOW) to PBSC. The contract expires 20 ------- September 2003, and we have been advised by the program office that there will be no follow-on contract for these services. We will however ensure that where feasible, task orders are PBSC. Page 7. Finding: Contracts That Were Prescriptive In our efforts to transition to PBSC some of our contracts may have been prescriptive; however, we disagree with the following: Contract 68-C-00-189 and Contract D-98-169 for Security Guard Services. We disagree with the findings and recommend that these contracts be deleted from table 2-3 as being contracts that were prescriptive. Since the events of September 11th we had to modify our guard services contracts to clearly specify our requirements. We no longer consider these services to be low risk, nor do we believe that the requirements restrict the contractor from using innovative approaches or shifting responsibility. Contract 68-W-02-020 Property Management Services. We disagree that this contract contained PBSC attributes which were overly prescriptive and recommend that this contract be deleted from the list of contracts identified as being prescriptive. The SOW included a brief background and purpose, then provided generalized support requirements, further delineated by specific performance objectives/appropriate performance measures/performance standards. Although we included a description of the labor categories, we do not believe that this impedes the contractor s ability to offer innovative approaches to determine how best to perform. Page 7/8. Finding: Incentives and Disincentives not Meaningful. Contract 68-W-00-104 Security Support Services. We agree with the OIG that this contract, awarded in June 2000, contains numerous disincentives of nominal dollar value. Working with the program office, we will review all of the incentives and disincentives in this contract and revise them as appropriate to be more meaningful. By June 30, 2003, we will ensure that the revised incentives and disincentives are incorporated into the contract for option period two. Page 14 Finding: Performance-Based Task Orders Issued Under Non-Performance-Based Contracts Not Shown as PBSC in FPDS. We agree that some data was not properly coded as PBSC or non-PBSC. Corrections have now been made to ICMS data, based on a review of ICMS data by OAM s operations divisions. Those changes have been re-submitted to FPDS and have been processed. We have not yet seen the results. We also agree that PBSC task orders under non-PBSC contracts were not being submitted as PBSC to FPDS. This discrepancy accounted for mostof the under-reporting of PBSC actions to FPDS. EPA has corrected its reporting process and has resubmitted the PBSC task order actions to FPDS. Due to a six week shutdown of FPDS, the re-submission has just been processed. Again, we have not yet seen the results. 21 ------- Page 15 Finding: Reporting Improvements Needed. The report states: Until EPA ensures that ICMS data is consistent with FPDS, the Agency s credibility as well as any subsequent stakeholder decision making could be adversely affected. ICMS data changes daily, while FPDS is currently updated only quarterly. One of the E-Gov initiatives planned for FY2004 is development of a new and better data gathering system, Federal Procurement Data System - Next Generation (FPDS-NG). This system will be updated more frequently - possibly overnight, but a final decision has not been made. After FPDS-NG is implemented, it is more likely that the two systems will more closely match. It should also be noted that EPA s current systems do not meet our business needs. They provide inadequate reporting, cumbersome tracking status, and redundant data entries which result in extraordinary inefficiencies and errors. We need a system that can interface with Government- wide systems as described in the ACQUISITION SYSTEM Project proposed in the FY 2005 OAM Capital Planning for Investment Control Process (CPIC). The proposed system will be a secure web-based tool, eliminate duplicate data entry, track actual costs and progress in real time, and provide a Commercial Off The Shelf (COTS) application consistent with the President s E-Gov initiatives. We appreciate the OIG s help in soliciting support for this system. RECOMMENDATIONS The OIG recommends that the Assistant Administrator for Administration and Resources Management, in coordination with the Senior Resource Officials for each Headquarters and regional office and other field locations promote maximum use of PBSC by: Recommendation 2-1 OIG: Developing and implementing a PBSC plan that includes short-term and long-term strategies for maximizing the use of performance based contracting. The plan should establish goals for PBSC use in program offices and monitor progress against those goals. OARM s Response: We concur with this recommendation and will provide a plan of action to comply with the recommendation in the final report. It would be helpful if the IG would assist us in soliciting support from senior officials in the Agency to engage AAs and RAs in helping the Agency meet its goals. Recommendation 2-2 OIG: Developing and implementing procedures to improve the use of PBSC, such as deploying experienced PBSC contracting personnel to assist program offices in developing performance- based statements of work, and ensuring that the statements of work contain the required elements. 22 ------- OARM s Response: Although we concur with this recommendation, we currently provide experienced PBSC contracting personnel to assist program offices, develop PBSC SOW templates, etc. We will provide details in the final report. Recommendation 2-3 OIG: Identifying forums, such as the Innovation Action Council, to help further promote PBSC, and to discuss PBSC best practices and issues on a regular basis. OARM s Response: We concur with the recommendation. We currently engage in several venues to promote PBSC. We solicit the IG support in getting buy-in at the senior level. Recommendation 2-4 OIG: Developing and implementing a procurement policy addressing the FAR Subpart 7.105 requirement to implement performance-based contracting methods, or provide a rationale for not using those methods. OARM s Response: We concur with this recommendation. The OIG recommend that the Assistant Administrator for Administration and Resources Management require the Director, Office of Acquisition Management to: Recommendation 3-1 OIG: Periodically review the accuracy of the contracting data contained within ICMS, including service codes, and make necessary corrections in a timely manner. OARM s Response: We concur with this recommendation. Please see comments referenced under the findings on page 14. Recommendation 3-2 OIG: Periodically compare ICMS and FPDS data to ensure the reliability of the information, and examine any major differences and correct in a timely manner. OARM s Response: We concur with this recommendation. Recommendation 3-3 OIG: Summarize the steps taken or planned, and progress made in correcting the PBSC data quality issues in OARM s Federal Manager s Financial Integrity Act report. OARM s Response: We concur with this recommendation and will address this and other data quality issues in the Federal Manager s Financial Integrity Act report. 23 ------- 24 ------- Appendix C Report Distribution Assistant Administrator for Administration and Resources Management Assistant Administrator for Environmental Information Assistant Administrator for Solid Waste and Emergency Response Assistant Administrator for Air and Radiation Assistant Administrator for Water Assistant Administrator for Prevention Pesticides and Toxic Substances, Assistant Administrator for International Affairs Assistant Administrator for Research and Development Agency Followup Official Agency Audit Followup Coordinator Associate Administrator for Congressional and Intergovernmental Relations Associate Administrator for Public Affairs Director, Office of Acquisition Management Director, Office of Emergency and Remedial Response Audit Liaison, Office of Administration and Resources Management Audit Liaison, Office of Environmental Information Audit Liaison, Office of Solid Waste and Emergency Response Inspector General (2410) Regional Administrators, Regions 1 through 10 Regional Audit Followup Coordinators, Regions 1 through 10 25 ------- |