ฃ
<
33
O
\
O *
2E I
U.S. ENVIRONMENTAL PROTECTION AGENCY
OFFICE OF INSPECTOR GENERAL
Improved Contract
Administration Needed for
the Customer Technology
Solutions Contract
Report No. 13-P-0398
September 16, 2013
Scan this mobile
code to learn more
about the EPA OIG.

-------
Report Contributors:	Michael D. Davis
Teren Crawford
Nancy Dao
Kevin Haas
Debra C off el
Abbreviations
CO
Contracting Officer
COTR
Contracting Officer Technical Representative
CPARS
Contractor Performance Assessment Reporting System
CSIRC
Computer Security Incident Response Capability
CTS
Customer Technology Solutions
EPA
U.S. Environmental Protection Agency
EPAAR
Environmental Protection Agency Acquisition Regulations
FAR
Federal Acquisition Regulations
ISO
Information Security Officer
NIH-CPS
National Institutes of Health's Contractor Performance System
OAM
Office of Acquisition Management
OARM
Office of Administration and Resources Management
OIG
Office of Inspector General
OMB
Office of Management and Budget
OTOP
Office of Technology Operations and Planning
PO
Project Officer
PPIRS
Past Performance Information Retrieval System
QAM
Quality Assurance Manager
QASP
Quality Assurance Surveillance Plan
SP
Service Provider
SRI
Service Request Interface
WCF
Working Capital Fund
Hotline
To report fraud, waste, or abuse, contact us through one of the following methods:
email: OIG Hotline@epa.gov	write: EPA Inspector General Hotline
phone: 1-888-546-8740	1200 Pennsylvania Avenue, NW
fax:	202-566-2599	Mail code 2431T
online:
http://www.epa.gov/oiq/hotline.htm
Washington, DC 20460

-------
^ฃDsrx
• A v
lSi
U.S. Environmental Protection Agency
Office of Inspector General
At a Glance
13-P-0398
September 16, 2013
Why We Did This Review
The U.S. Environmental Protection
Agency Office of Inspector
General conducted this audit to
determine whether the EPA
implemented effective contract
administration for its Working
Capital Fund contract EPW08034.
The Office of Administration and
Resources Management's Office
of Acquisition Management and
the Office of Environmental
Information's Office of Technology
Operations and Planning perform
contract administration activities
with contracting officers, project
officers and quality assurance
managers for the Customer
Technology Solutions contract
EPW08034. Contract
administration involves those
activities performed by
government officials after a
contract has been awarded to
determine how well the
government and the contractor
performed to meet the contract
requirements. Contract
administration assists the
government in assuring it receives
the goods and services for which it
paid.
This report addresses the
following EPA theme:
• Embracing EPA as a high
performing organization.
For further information, contact
our Office of Congressional and
Public Affairs at (202) 566-2391.
The full report is at:
www.epa.qov/oiq/reports/2013/
20130916-13-P-0398.pdf
Improved Contract Administration Needed for the
Customer Technology Solutions Contract
What We Found
Based on our review of the WCF contract EPW08034, which ended
September 2012, the EPA needs to improve its contract administration to
assist in managing other similar type contracts. The EPA did not, as stated by
Office of Management and Budget, Federal Acquisition Regulations and
agency guidelines:
•	Use performance standards to measure cost outcomes.
•	Complete any of the required contractor performance evaluation reports.
•	Maintain required contract administration documents.
The EPA did not have policies in place that would require performance metrics
and standards to be linked to cost outcomes and procedures to ensure
contract administrators maintain sufficient documents in the official contract
files. The EPA did not complete contractor performance reports because of
insufficient guidance and inadequate communications during contractor
performance system changes and personnel reassignments. The EPA's
contractor performance systems include the National Institutes of Health's
Contractor Performance System and subsequently the Contractor
Performance Assessment Reporting System, which transmits information into
the Past Performance Information Retrieval System. The EPA's ineffective
contract administration may have hindered the ability of EPA staff to ensure
that the contractor successfully met agency needs, as well as its ability to
determine whether the EPA achieved the best value for the $85 million
expended on the WCF contract.
Recommendations and Planned Agency Corrective Actions
We recommend that the OARM assistant administrator update its policies and
procedures to ensure that contract performance metrics and standards link to
cost outcomes. We also made recommendations involving updating internal
programs to provide oversight and accountability for linking metrics to cost
outcomes and the review and submission of contractor performance
evaluation reports for the contract reviewed. Further, we recommend that
OARM develop contract administration procedures related to the transfer of
documents when reassigning contract administrative staff.
OARM concurred with four of the six recommendations and did not concur
with recommendations 1 and 2. The OARM's corrective actions for
recommendations 3 through 6 do not contain milestones nor completely
address the recommendations and are unresolved.
Noteworthy Achievements
The Office of Acquisition Management implemented the Balanced Scorecard
program in fiscal year 2011 for the EPA Acquisition Systems to provide the
necessary checks and balances to ensure procurement documents are of the
highest quality and comply with applicable laws, regulations and policies.

-------
^ED sx
ฆ ฃ% '
< rJK-7 5	UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
| W\/C/ ฎ	WASHINGTON, D.C. 20460
"U PRO^
THE INSPECTOR GENERAL
September 16, 2013
MEMORANDUM
SUBJECT: Improved Contract Administration Needed for the
Customer Technology Solutions Contract
Report No. 13-P-0398
FROM: Arthur A. Elkins Jr. /fiM*f L* ฆ (/^r^
TO:	Craig E. Hooks, Assistant Administrator
Office of Administration and Resources Management
Renee P. Wynn, Acting Assistant Administrator and Chief Information Officer
Office of Environmental Information
This is our report on the subject audit conducted by the Office of Inspector General of the
U.S. Environmental Protection Agency. This report contains findings that describe the problems the
OIG has identified and corrective actions the OIG recommends. This report represents the opinion of the
OIG and does not necessarily represent the final EPA position.
Action Required
The agency agreed with four of the six recommendations in this report. However, all six
recommendations are considered unresolved pending our receipt of EPA's corrective action plan and
estimated completion dates that correspond with the specific recommended actions. Therefore, for all
recommendations, you are required to provide a corrective action plan with planned completion dates within
60 days of report issuance.
We will post this report to our website at http://www.epa.gov/oig.
If you or your staff have any questions regarding this report, please contact Richard Eyermann, acting
assistant inspector general for the Office of Audit, at (202) 566-0565 or evermann.richard@epa.gov;
or Mike Davis, director for Efficiency Audits, at (513) 487-2363 or davis.michaeld@epa.gov.

-------
Improved Contract Administration Needed for the
Customer Technology Solutions Contract
13-P-0398
Table of C
Chapters
1	Introduction		1
Purpose		1
Background		1
Noteworthy Achievements		3
Scope and Methodology		4
2	Ineffective Administration of Contractor Performance		6
OMB, FAR and EPAAR Provide Guidance for Managing Contractors		6
Performance Metrics and Standards Not Linked to Cost Outcomes		7
Contractor Performance Evaluation Reports Not Completed		9
Conclusion		11
Recommendations 		11
Agency Comments and OIG Evaluation		12
3	Inadequate Contract Records Management		13
Federal and Agency Guidance on Records Management		13
Agency Official Contract Files Missing Required Documents		14
Conclusion		16
Recommendations 		16
Agency Comments and OIG Evaluation		16
Status of Recommendations and Potential Monetary Benefits		17
Appendices
A Contract Performance Requirements Summary		18
B PO, CO and QAM Reassignments on CTS Contract		22
C Relevant Recordkeeping Criteria		23
D Agency Comments on Draft Report		25
E Distribution		32

-------
Chapter 1
Introduction
Purpose
The U.S. Environmental Protection Agency Office of Inspector General
conducted this audit to determine whether the EPA implemented effective
contract administration for its Working Capital Fund contract EPW08034.
Even though the contract ended on September 30, 2012, the EPA awarded
subsequent contracts for the same services. In addition, the EPA can apply
lessons learned from this audit to improve its contract administration that will
assist in managing other similar types of contracts (see chapters 2 and 3).
Background
Contract administration involves government activities performed after an agency
awards a contract to determine whether the contractor met contract requirements.
Contract administration activities include controlling costs, monitoring,
measuring and reporting contractor performance, and maintaining records in the
official contract files. Contract administration constitutes that primary part of the
procurement process that assures the government gets what it pays for.
The WCF is a revolving fund authorized by law to finance a cycle of operations in
which federal agencies can charge the costs for goods or services provided to its
customers. WCF customers include the EPA's program offices and regions. The
role of the WCF is to provide a centralized source of administrative and support
services for the EPA. The WCF strives to reduce the costs of services in the
agency through improved efficiencies gained by achieving economies of scale,
greater consumer bargaining power, and reduction in overhead.
The Office of Administration and Resources Management's Office of Acquisition
Management and the Office of Environmental Information's Office of
Technology Operations and Planning provide contract administration on the WCF
contract EPW08034. OTOP personnel manage the work performed under the
contract, which is with Customer Technology Solutions. CTS, a WCF data
processing service, provides and coordinates all information technology end-user
support and services for the EPA's headquarters and field offices. Critical
participants in the contract administration of the WCF contract include:
•	OAM's contracting officers, who have overall responsibility for the contract.
•	OTOP's project officers, quality assurance managers and quality assurance
evaluators, who monitor contractor performance and customer satisfaction.
13-P-0398
1

-------
Even though the Office of the Chief Financial Officer manages the WCF,
including reviews and processing of invoices, OAM and OTOP provide contract
administration functions.
In April 2008, the EPA awarded an indefinite delivery/indefinite quantity hybrid
contract. This hybrid contract also contained provisions for performance-based
award fees and fixed hourly rates and material costs for a time and materials
contract. This contract required the contractor to provide technology services to
approximately 12,000 program office users scattered throughout 16 states and the
District of Columbia. The total potential value of the contract was over
$200 million and covered the contract base period of June 1, 2008, to
September 30, 2012, and an option period available through December 31, 2016.
The EPA decided not to exercise the option period because the contractor had not
met the required performance metrics. Figure 1 summarizes the WCF's top five
data processing service contracts obligated as of May 2012.
Figure 1: Top Five WCF obligations as of May 20121
WCF Contracts (millions of obligations)*
ฆ	Network Security and
New Custom Application
ฆ	Cominercial-Off-The-
Shelf
ฆ	Customer Technology
Solutions
ฆ	Infrastructure
ฆ Hosting
Source: OIG analysis of EPA data.
The agency identified approximately $72 million of funds obligated for the CTS
contract EPW08034 as of May 2012. By October 2012, the EPA's Compass
Financial2 system reported that the agency's expenditures for this CTS contract
increased to approximately $85 million.
1	OTOP's WCF coordinator provided this information on May 11, 2012, in the form of an email.
2	Compass is the EPA's official financial database.
13-P-0398
2

-------
The Federal Acquisition Regulation, Subsection 37.601(b), requires that:
Performance-based contracts for services shall include—(1) A
performance work statement (PWS); (2) Measurable performance
standards (i.e., in terms of quality, timeliness and quantity, etc.) and
the method of assessing contractor performance against performance
standards; and (3) Performance incentives where appropriate.
The FAR Subsection 37.603(a) requires that performance standards establish the
performance level required by the government to meet the contract requirements.
In addition, this same FAR Subsection requires the performance standards to be
measurable and structured to permit an assessment of the contractor's
performance. Th eEPA Contracts Management Manual [CMM], Section 11.1.5.7,
"Developing a Performance Based Service Contract Quality Assurance
Surveillance Plan," states that a:
The quality assurance surveillance plan (QASP) defines what the
Government must do to ensure that the contractor has performed
in accordance with the PWS performance standards.
The QASP objective is to assess the ability of the contractor to achieve the
defined performance standards and fulfill contractual obligations. According to
the CTS contract, the EPA defined the performance standards in the contract's
Technical Exhibit under the title "Performance Requirement Summary."
This CTS performance-based contract has a performance work statement,
measurable performance standards, performance incentives and a QASP. The
Performance Requirement Summary identified 44 performance metrics and
standards. According to the QAM, the EPA used no other performance metric or
standard to measure contractor performance. In addition, the EPA performed
weekly and monthly evaluations based on the 44 performance metrics to monitor
contractor performance per the FAR. According to the contract's award fee plan,
the EPA also used the 44 performance metrics and standards to determine whether
the EPA would provide an award fee to the contractor. The 44 performance
metrics identified categories that relate to quality and technical performance
(a degree of excellence), timeliness and scheduling (meeting milestones), and
business relations (customer satisfaction). Details are in appendix A.
Noteworthy Achievements
OAM implemented the Balanced Scorecard Program in fiscal year 2011 for the
EPA Acquisition Systems. The internal control program works in conjunction
with the Balanced Scorecard Program to provide the necessary checks and
balances to ensure that procurement documents are of the highest quality and in
compliance with applicable laws, regulations and policies. The Balanced
Scorecard Program also seeks to include measures of quality, cost, timeliness,
customer service and employee alignment with skills, to provide an in-depth,
13-P-0398
3

-------
predictive performance management system. Even though the EPA implemented
specific measures in the Balance Scorecard Program to ensure that past
performance information is timely and included in the Contractor Performance
Assessment Reporting System, the specific measures did not start until the end of
fiscal year 2012.
Scope and Methodology
We conducted this performance audit in accordance with generally accepted
government auditing standards. Those standards require that we plan and perform
the audit to obtain sufficient, appropriate evidence to provide a reasonable basis
for our findings and conclusions. We believe that the evidence obtained provides
a reasonable basis for our findings and conclusions based on our audit objectives.
We conducted the audit from May 2012 to June 2013 at OTOP, OAM and the
Office of the Chief Financial Officer's Research Triangle Park Finance Center.
During the audit, we obtained the top five WCF data processing service contracts
and reviewed the contract with the highest reported obligated amount to
determine whether the EPA was effectively managing WCF contracts. We limited
our review to auditing the EPA's contract administration for the CTS EPW08034
contract because it accounted for approximately $72 million (60 percent) of the
approximately $120 million in the EPA WCF contracts (see figure 1).
We reviewed memoranda, regulations, guides and other documents to gain an
understanding of the requirements and processes used to provide effective
contract administration. The specific documents reviewed included:
•	Office of Management and Budget memoranda.
•	Office of Federal Procurement Policy Memoranda, and Guides.
•	FAR Parts 1, 4, 16, 32, 37, 42 and 46.
•	Environmental Protection Agency Acquisition Regulations Sections
1509.170 and 1542.15
•	The EPA CMM.
Appendix C includes relevant recordkeeping criteria.
We looked at OMB, FAR, agency polices and guidance for contract and contract
administration requirements. We discussed with OAM, OTOP and Research Triangle
Park Finance Center staff their procedures used to manage contractor performance,
invoices, monthly progress reports and other needed contract administration
functions used for effective internal controls. We also discussed other background
information that pertains to WCF procedures and associated key players. The WCF
background information obtained involved the systems used, documents managed
and other WCF funding procedures used. Other WCF key players included OTOP's
Director of Customer Business Support Staff, who oversees WCF workload,
collections, billings and reports. The director also interacts with WCF customers who
13-P-0398
4

-------
provide feedback on service quality. We reviewed contract invoices, checklists,
monthly progress reports, QASPs, prior audit reports, award fee determination
reports, as well as other supporting documentation, to determine whether the EPA
conducted effective contract administration.
Prior Agency OIG Reports
The EPA OIG issued the following reports with findings related to CTS contract
administration:
Report No. 10-P-0194, EPA Needs to Improve Management Practices to
Ensure a Successful Customer Technology Solutions Project, August 23, 2010.
The agency's lack of acquisition planning led to questions involving:
•	The quality of the helpdesk supporting the project.
•	A quality management program that was not finalized.
•	Key business processes to support ongoing operations not being defined.
•	Vacant leadership positions needed to facilitate communication and
coordination with customers about CTS equipment deployments.
We did not follow up on the contractor providing helpdesk support on the project
because the EPA cancelled the contract. We did follow up on areas that affected
contract administration. We found the following:
•	The EPA finalized the QASP.
•	The EPA ensured the contractor defined key business process documents.
•	The EPA filled vacant leadership positions.
Report No. ll-P-0705, EPA Contract Oversight and Controls Over Personal
Computers Need Improvement, September 26, 2011. The EPA paid the CTS
contractor a total of $489,734 over an 11-month period for 3,343 seats. A standard
seat includes a leased computer with accessories and technical support, which was
not ordered by the agency during the aforementioned period. We recommended that
the EPA should improve controls for updating data in the fixed assets database, and
should retain property acquisition documentation in accordance with retention
requirements. In addition, we recommended that the agency should have a
separation of duties in its property staff positions and consider assigning permanent
property positions. We did not follow up on this condition because the EPA
decided not to exercise the contract option award terms because the contractor had
not met the required performance metrics. However, during this review, we found
similar internal control issues in terms of updating databases, including:
•	Retaining appropriate documentation.
•	Maintaining permanent contract positions (see the following chapters).
13-P-0398
5

-------
Chapter 2
Ineffective Administration of Contractor Performance
OAM did not effectively evaluate cost outcomes and report any of the four
required annual contractor performance evaluation reports in the Past
Performance Information Retrieval System between September 2009 and
March 2013. OMB and FAR require agencies to assess cost outcomes, while the
FAR and EPAAR require agencies to report on contractor performance
information in the PPIRS. Deficiencies occurred because of insufficient guidance
and inadequate communications during contractor performance system changes
and personnel reassignments. Consequently, the EPA did not effectively monitor
costs to ensure that the contract, with expenses at approximately $85 million,
achieved the best possible cost outcomes and value. In addition, the EPA did not
provide other federal acquisition community users with contractor past
performance data so that other federal government agencies can make effective
management decisions on whether to conduct business with the same contractor.
OMB, FAR and EPAAR Provide Guidance for Managing Contractors
OMB and the FAR established requirements for evaluating contractor
performance for cost outcomes. OMB policies state that evaluation factors be
directly linked to cost performance results.3 Another OMB policy4 addresses the
evaluating of cost controls based on the contractor's:
•	Ability to perform within or below budget.
•	Use of cost efficiencies.
•	Relationship of negotiated costs to actual costs.
•	Submission of reasonably priced change proposals.
•	Providing current, accurate and complete billing in a timely manner.
This policy addressed the assessing of the contractor's effectiveness in
forecasting, managing and controlling contract cost. This policy also stated cost
controls are needed for fixed price contracts. The FAR5 requires that, for incentive
fee type and time and materials contracts, evaluating factors be linked to cost
performance results and cost controls, respectively.
The FAR and EPAAR regulations establish requirements for reporting contractor
performance information. Specifically, the FAR6 requires that agencies prepare an
annual evaluation of contractor performance that closely parallels the award fee
3	OMB memorandum. [ppropriale Use of Incentive Contracts, December 4, 2007; and OMB memorandum.
Improving Government Acquisition, dated July 29, 2009,
4	OMB policy Best Practices for Collecting and Using Current and Past Performance Information, May 2000.
5	FAR Subsections 16.401(e)(2), 16.402-l(a) and 16.601(c)(1).
6	FAR Subpart 42.1500, Subsections 42.1503(b), and 42.1503(c).
13-P-0398
6

-------
determinations and submit past performance information electronically to the
PPIRS in accordance with agency procedures. The agency's acquisition
regulations7 require the agency to use the National Institutes of Health's
Contractor Performance System, subsequently replaced by CPARS, to record
contractor performance information. The systems collect and transmit information
into the PPIRS.
Performance Metrics and Standards Not Linked to Cost Outcomes
The EPA's 44 performance metrics and standards identified in the Performance
Requirements Summary for the EPW08034 CTS contract did not include cost
outcomes as required by the FAR and addressed in OMB guidance. The EPA used
the 44 performance metrics and standards to evaluate contractor performance on
the CTS contract and determine whether the contractor would receive an award
fee. The QASP identifies the procedures used by the QAM and quality assurance
evaluator to review contractor performance based on the Performance
Requirement Summary and its 44 performance metrics and standards. The QAM
assembled biannual reports that identified and rated each of the 44 performance
metrics and standards used to determine whether the contractor would be granted
an award fee. Appendix A identifies the 44 metrics and standards that relate to
quality, schedule and customer satisfaction outcomes. The EPA did not include
any metrics and standards relating to cost outcomes as required by the FAR. The
44 performance metrics and standards did not include the OMB-identified
definitions for cost controls that cover:
•	The contractor's ability to perform within or under budget.
•	Use of cost efficiencies.
•	Relationship of negotiated costs to actual costs.
•	Submission of reasonably priced change proposals.
•	The contractor providing current, accurate and complete billing in a
timely manner.
Also, the CTS contract performance work statement identified a requirement for
the contractor to analyze potential cost savings from new technologies and
business process changes. FAR Subsection 37.602(b) (3) requires agencies to
"rely on the use of measurable performance standards and financial incentives in a
competitive environment to encourage competitors to develop and institute
innovative and cost-effective methods of performing the work." The EPA did not
use the 44 performance metrics and standards or incentives to encourage
competitors to develop and institute innovative and cost-effective methods of
performing the work per FAR requirements.
As part of the EPA Balanced Scorecard Program approach, OAM identified
measures for cost control in contractor performance evaluations as principles
7 EPAAR Subpart 1542.1503(a) (CPARS), which replaced 1509.170-3(c) and 5(c) (NIH-CPS).
13-P-0398
7

-------
needed to improve its acquisition process. However, the EPAAR and CMM did
not require that for incentive fee and time and material type contracts the
performance evaluation factors include cost outcomes. In accordance with the
December 2007 OMB memorandum, chief acquisition officers and senior
procurement executives review agency's acquisition policies to ensure that incentive
fees are linked to acquisition outcomes such as cost, schedule, and performance
results. Current EPA policy does not address this requirement. Even though this
contract has ended, having this requirement will ensure that existing and future
contracts will contain the required cost outcome metrics and standards.
By not linking the award fee to cost outcomes as required, the EPA did not
effectively hold the contractor accountable for cost inefficiencies and
inaccuracies. We found examples where the contractor billed for services that
were not rendered. For example, Table 1 identifies examples of 13 invoices
totaling nearly $1 million that the agency suspended for inaccurate billings.
Suspended contractor invoice payments due to inaccurate billings indicate
potential problems exist with contract cost.
Table 1: Invoices suspended
Invoice
Suspended
amounts
Bill date
Reason for suspension
1
$2,734.41
07/20/10
Units not deployed. Invoice is inaccurate
(data does not support the billable amount
for the time period).
2
3,125.04
10/26/10
Units not deployed.
3
117,775.00
08/13/09
Dispute over number of printers installed.
4
4,330.57
08/21/09
Fifteen student/intern seats deployed on a
Saturday. User cannot accept deployment on
Saturday. Need to count these seats next
month.
5
281,050.00
10/13/09
Dispute over printers installed.
6
391,650.00
11/20/09
Dispute over printers installed.
7
3,767.62
08/12/10
Units not deployed.
8
53,082.25
02/27/12
Work not completed
9
15,961.82
11/13/12
Typo error on prior payment.
10
29,784.47
05/21/12
This invoice indicates a double billing for the
same service.
11
9,674.35
08/12/10
Units not deployed.
12
130.21
10/22/10
Units not deployed.
13
27,433.72
07/06/12
Questioned billing method of rounding.
Total
$940,499.46


Source: OIG data analysis.
In addition, since FAR requires cost outcomes from past contractor performance
be inputted into PPIRS, the EPA should have evaluated the cost outcome based on
contractor performance standards. As a result, the EPA cannot sufficiently inform
other federal agencies about the contractor's cost performance.
Even though the agency did not award the contractor an award fee, there could
have been additional corrective actions implemented by having cost outcomes tied
13-P-0398
8

-------
to contractor performance standards. This would have provided additional
enforcement measures to prevent recurring or escalating problems. According to
the QASP, the QAM has the responsibility to monitor the contractor's
performance to control cost and take corrective actions. The performance work
statement requires the contractor to look for cost savings through enhanced
technology. The EPA did not tie the requirement to performance standards so the
EPA did not use this requirement for determining an award fee. Including cost
outcomes could have driven the overall ratings up if the contractor identified
savings and could have driven the overall ratings further down if the contractor
did not identify savings. Such actions would facilitate a determination as to
whether the contractor is performing to meet the overall contract needs and
providing the most value for the government.
Contractor Performance Evaluation Reports Not Completed
The EPA did not complete the four required annual contractor past performance
evaluation reports. The EPA completes contractor past performance evaluations
when an agency official enters the finalized reports in CPARS. Prior OIG reports
identified similar issues where the EPA did not complete the evaluations or report
past performance information into required systems in a timely fashion.8 Both the
FAR and EPAAR require the agency to conduct and report the evaluations in the
past performance system after the end of each 12-month contractor performance
period. The initial EPAAR Subpart 1509.170, dated 2002, required the EPA to
use the NIH-CPS to report past performance information within 90 business days
from the date the contracting officer initiates the evaluation. As of September
2010, the new EPAAR Subsection 1542.1502 replaced Subpart 1509.170, and
requires the use of the Department of Defense's CPARS [see prior Footnote 7],
CPARS requires reporting of past performance information within 120 days.
Table 2 identifies the specific contractor past performance data not timely entered
into the system.
Table 2: Timeliness of Contractor Performance Evaluations for EPW08034
Evaluation period
Reporting
system *
Evaluation
Date due
Interim/final
evaluations
# Calendar Days
Past Evaluation
Due Date
(04/01/2013=OIG
cut-off date)
10/01/08-09/30/09
NIH-CPS
02/11/10 (1)
Not prepared
1145
10/01/09-09/30/10
CPARS
02/11/11 (1)
Not prepared
780
10/01/10-09/30/11
CPARS
02/10/12 (1)
Not prepared
416
10/01/11-09/30/12
CPARS
01/28/13 (2)
Not prepared
63
*Past performance information in NIH-CPS/CPARS is automatically transmitted to the PPIRS.
(1) 95 business days allowed for years 1-3's evaluation process. (2) 120 calendar days allowed for
evaluation process for year 4 per new EPAAR Subsection 1542.1502.
8 Report No. 10-R-0113, EPA Should Improve Its Contractor Performance Evaluation Process for Contractors
Receiving Recovery Act Funds, issued April 26, 2010; and Report No. 12-P-0417, Weaknesses in EPA's
Management of the Radiation Network System Demand Attention, issued April 19, 2012.
13-P-0398
9

-------
Source: OIG analysis of contractor performance evaluations as of 04/01/2013.
The 44 performance metrics and standards (see appendix A) identify the
performance categories quality, timeliness and business relations. Quality
assurance evaluators review contractor performance based on the metrics and
standards and submit semiannual reports that contain needed information to input
into the CPS or CPARS, but the EPA did not use such information to complete
contractor past performance evaluation reports in the systems.
The EPA did not have the contractor performance information in PPIRS because
of inadequate communication during the NIH-CPS and CPARS transitions and
personnel reassignments. NIH-CPS and CPARS are the official systems used by
the agency to download past performance information into PPIRS. Even though
the FAR required contractor performance information to be entered into PPIRS,
the EPA did not provide clear guidance to enforce the reporting of contractor
information for existing contracts. The OAM intranet website stated that the
CPARS would be used for new solicitations issued on October 3, 2011, or later.
In addition, OAM's Intranet also emphasized EPAAR changes for new
solicitations but did not require the EPA to input contractor performance
information into CPARS for existing contracts. The Interim Policy Notice 10-03
issued by OAM in March 2010 also emphasizes that solicitations issued, contracts
awarded and options exercised after May 14, 2010, are to follow the EPAAR
deviations to report contractor performance information in CPARS. The EPAAR
deviations did not require existing contracts to report contractor performance
information. Since the CTS contract, awarded in 2008, was an active contract at
the time and EPA did not provide clear guidance on existing contracts, the EPA
did not report on four contractor performance evaluation periods (see table 2
above).
In addition, when the EPA's OAM and OTOP reassigned staff, these offices did
not communicate adequate procedures to ensure reporting of contractor
performance information. The initial QAM stated she asked the CO about putting
information into NIH-CPS, but the CO said that, since the EPA changed systems,
they should do nothing. One PO, assigned to the contract for a few months, did
not fully understand the CPARS assessment process and did not have access to
the CPARS. One CO stated that she transitioned out before the end of the 12-
month evaluation period. Another CO said she requested the PO to assess the
contractor performance but received no response from the PO. With many
personnel reassignments (see appendix B), the EPA has no effective procedure to
communicate what CPARS evaluations have or have not been done. The
Department of Defense CPARS policy guide9 states an interim evaluation in
CPARS is required upon a significant change within the agency provided 6
months of performance has occurred. The changes include change in program or
9 The PPIRS website, www.ppirs.gov, and OMB memorandum, Improving Contractor Past Performance
Assessments: Summary of the Office of Federal Procurement Policy's Review, and Strategies for Improvement,
dated January 2011, acknowledges the DOD policy as guidance for federal agencies on contractor performance
information. The DOD policy was subsequently revised in June.
13-P-0398
10

-------
project management responsibility and transfer of a contract to a different
contracting activity. It also states that an interim evaluation in CPARS shall be
started prior to transfer of assessing official duties from one individual to another,
to ensure continuity.
If COs did not report the contractor performance information in the past
performance systems for all contract performance periods, performance reports
are not available to inform the EPA and other federal agencies about the
contractor's performance. According to the OMB memorandum identified in the
footnote 9, this information is used in the source selection process to assist other
agencies in assessing past performance when making future acquisitions. Without
timely availability of contractor past performance information that includes
information on contract cost, the EPA and other federal agencies may
inadvertently award contracts to contractors that have a history of poor
performance on current or past government contracts. Since the EPA did not
provide an award fee throughout the entire life of this 4-year performance-based
contract, there may be concerns with the contractor performance that should be
reflected and available for consideration by other agency offices and government
agencies. An OMB memorandum, Clarifying Chief Acquisition Officer Roles and
Responsibilities, dated October 2012, states that chief acquisition officers ". . .
should lead efforts to, among other things, improve the value of contractor past
performance assessments and increase the transparency of contractor business
integrity data so that the Federal Government only does business with reputable
firms."
Conclusion
Cost controls and contractor performance information have been the focus of
OMB memoranda and the Balanced Scorecard Program addressed in chapters 1
and 2 above. The EPA did not evaluate cost outcomes and report on any required
contractor performance information as required. Having effective cost controls
and past contractor performance information for this contract valued at
approximately $85 million will help the EPA assure that the government gets
what it paid for. It will also provide transparency so that other federal government
agencies can make effective management decisions on whether to conduct
business with the same contractor.
Recommendations
We recommend that the assistant administrator, Office of Administration and
Resources Management:
1. Update the policies to include procedures that would ensure that, for all
incentive and time and material contracts, contract performance metrics
and standards include cost outcomes.
13-P-0398
11

-------
2.	Update internal programs that provide oversight and accountability to
ensure that, for all incentive type and time and material contracts, the EPA
performance metrics and standards link to cost outcomes.
3.	Require OTOP, in conjunction with OAM, to review contractor
performance reports generated per the QASP. Require OTOP to
coordinate all actions taken with the COs and POs and require the COs to
input contractor performance information into CPARS.
4.	Require COs to input interim reports into CPARS prior to their
reassignments.
Agency Comments and OIG Evaluations
OARM did not concur with recommendations 1 and 2, but concurred with
recommendations 3 and 4. However, recommendations 3 and 4 are unresolved
pending receipt of corrective actions that correspond with the causes and
recommendations.
OARM disagreed with recommendations 1 and 2. OARM stated EPA does not have
to control cost or manage the contract-based cost incentives for this WCF contract
because the contract has fixed rates so it is a fixed unit price contract. FAR
Subsections 16.202-1 and 203-l(b) requirements state that for fixed price contracts,
the award fee or incentive is based solely on factors other than costs; however, this
contract's requirements and data indicated a contract type other than a fixed price
contract. FAR Subsection 16.201(b) states "time and material contracts and labor
hour contracts are not fixed price contracts." The EPA's Acquisition System and the
Federal Procurement Data System identified this contract to be a time and materials
contract. The contract also identified changes to fixed hourly rate and negotiated
hours throughout the contract 4-year time period and identified fixed rate and other
elements that is consistent with FAR Subpart 16.6 for time and materials contracts.
Recommendations 1 and 2 also reflect on the performance standards having cost
outcomes and the decision to award a fee based on cost outcomes. FAR Subsection
1.102-2 states that performance standards satisfy the customer in terms of cost,
quality and timeliness of delivered product or service, so performance standards as
well as the decision to award a fee should link to cost outcomes.
OARM stated during the exit conference that issues found and addressed under
recommendations 3 and 4 relate to inadequate management oversight. Based on the
causes addressed above, we found no EPA procedures in place to direct the actions
of inputting interim performance information into CPARS when there are staff
reassignments. This detailed direction would assist management in providing the
oversight to ensure the contractor performance information is input into CPARS. In
addition, this is a recurring issue because prior audits recommended improvements
to management oversight that the agency agreed to yet the conditions still exist (see
footnote 8). As a result, we concluded that this further direction is needed.
13-P-0398
12

-------
Chapter 3
Inadequate Contract Records Management
The official contract files maintained by COs, POs and the QAM were missing
required official documents. These missing documents were:
•	Thirty-four of 48 monthly progress reports.
•	A post award conference meeting document.
•	Three of four CO annual contract invoice reviews.
•	A financial monitoring review report and the CO's applicable responses.
•	Initial or finalized QASPs.
•	Six of eight letters sent to the contractor providing its performance ratings.
FAR Subsection 4.802 and agency policies require that contract files be
maintained under central control to ensure effective documentation, safeguarding
and efficient retrieval. These required documents were missing due to ineffective
oversight of the policies and procedures in place to ensure the easy retrieval of
official documents during the many staff reassignments. Because the EPA did not
maintain adequate files on an approximately $85 million contract, the agency
could not provide assurance that the EPA made informed business decisions to
ensure contractors met contract requirements and the EPA complied with criteria.
As a result, the EPA cannot also provide assurance that the government
effectively got what it paid for.
Federal and Agency Guidance on Records Management
FAR Subsection 4.801, General, states documentation in the files shall be
sufficient for purpose of providing a complete background for informed decisions
and furnishing essential facts in the event of litigations or congressional inquiries.
FAR Subsection 4.802, Contract Files, paragraphs (c) and (d), states:
Files must be maintained at organizational levels that ensure:
(1) Effective documentation of contract actions; (2) Ready
accessibility to principal users; (3) Minimal establishment of
duplicate and working files; (4) The safeguarding of classified
documents; and (5) Conformance with agency regulations for file
location and maintenance. If the contract files or file segments are
decentralized (e.g., by type or function) to various organizational
elements or to other outside offices, responsibility for their
maintenance must be assigned. A central control and, if needed, a
locator system should be established to ensure the ability to locate
promptly any contract files.
13-P-0398
13

-------
The EPA policies and procedures emphasize the need for creating and managing
the records necessary to document the agency's official activities and actions. The
following agency policies are applicable.
•	EPA Classification No. CIO 2155, EPA Records Management Policy,
requires that all agency employees are responsible for creating, managing
and filing records for safe storage and efficient retrieval.
•	The EPA Records Management Manual maintains that agency staff must
capture records by filing, storing or otherwise systematically maintaining
them in a recordkeeping system to ensure information is accessible to all
authorized staff, including related records stored on special media or in
different locations.
•	EPA Record Series, Schedule 202, NARA Disposition No. Nl-412-06-
6/5, calls for COs and contracting officer representatives to work together
to determine who is responsible for maintaining specific documents to
minimize duplication while still providing an adequate audit trail (see
appendix C for more criteria).
Agency Official Contract Files Missing Required Documents
OAM COs and OTOP POs had inadequate records management based on FAR,
EPAAR and the EPA policy. The official contract file and files maintained by
each CO and PO were missing the following:
•	Thirty-four of 48 monthly progress reports.
•	A post award conference meeting document.
•	Three of four CO annual contract invoice reviews.
•	A financial monitoring review report and the CO's applicable responses.
•	Initial or finalized QASPs.
•	Six of eight letters sent to the contractor providing its performance ratings.
In addition, the EPA did not centrally control the documents that did exist, as
required by the FAR Subsection 4.802(d), as stated above. EPA's OAM
procedures did not establish a requirement for a central repository. The EPA's
various COs, POs and QAMs individually maintained selected documents. For
example, the COs did not maintain the finalized QASP in the official contract file.
The PO maintained the finalized QASP separately in their office computer. Of the
14 monthly progress reports received, the current CO provided only 11, one PO
provided two, and another PO provided one. These monthly progress reports
assist in determining whether the contractor met contract requirements. We asked
the current CO for contractor performance evaluations, but he could not provide
any. Two different COs provided six award fee evaluation reports. The evaluation
reports assist in determining whether EPA complied with FAR and EPAAR
requirements for measuring contractor performance.
13-P-0398
14

-------
The OIG has noted this problem with the agency's administration of its contracts
during multiple audits. The EPA's OAM Balanced Scorecard Program also
identified similar issues. Table 3 lists the OIG reports and the OAM document.
Table 3: Related audit reports
Report
Finding
EPA OIG 11-P-0705, EPA Contract
Oversight and Controls Over Personal
Computers Need Improvement
We found similar internal control issues in
terms of updating databases, retaining
appropriate documentation, and
maintaining permanent contract positions.
EPA OIG draft report, Improvements
Needed in EPA's Smartcard Program to
Ensure Consistent Physical Access
Procedures and Cost Reasonableness
Staff did not assure adequate data
maintenance and that incomplete contract
files resulted from issues within both the
project management and contract
management offices.
OAM Balanced Scorecard
An internal audit identified contract
documentation issues.
Source: OIG analysis.
The agency's inadequate record management persists because OAM and OTOP
did not have adequate oversight of policies and procedures in place to ensure the
easy retrieval of official documents during the many staff reassignments (see
appendix B). OAM and OTOP procedures did not hold responsible personnel
accountable for adequate records management during CO and PO personnel
reassignments. The CO and PO did not perform an inventory of required
documents to ensure the prior CO and PO maintained documentation. In addition,
OAM and OTOP did not implement procedures to require the transference and
maintenance of data in a centralized location or key locator system to identify
where documents are located during CO and PO personnel changes. The FAR
Subsection 4.802(d) states: "A central control and, if needed, a locator system
should be established to ensure the ability to locate promptly any contract files."
As a result of ineffective records management, files cannot be readily accessible
to principal users who are required to facilitate and ensure the effective
accomplishment of contract requirements and contract closeout. An overall
analysis of all documented monthly progress reports and annual invoice reviews
would have provided a better indication of the contractor achieving contract
requirements. In addition, the EPA may be unable to make effective management
decisions and provide reasonable assurance that the agency complied with
applicable regulations. If the CO centrally maintained all contractor performance
evaluations, the CO would have had past performance information readily
available to ensure compliance with FAR and EPAAR (see chapter 2).
Consequently, the EPA cannot reasonably make informed decisions based on
missing documents and ensure the government got what it paid for.
13-P-0398
15

-------
Conclusion
Without complete and organized contract files, the EPA is unable to make
effective management decisions and provide reasonable assurance that the
contractor met contract requirements and the EPA complied with applicable
regulations. Although the contract has ended, the EPA needs to maintain adequate
records to make effective and informed decisions on contractor past performance
for contract closeout purposes and in the event of litigation or congressional
inquiries. Since the agency continues to acquire those same services, records
would provide needed lessons learned to improve management of the new
contracts.
Recommendations
We recommend that the assistant administrator, Office of Administration and
Resources Management:
5.	Develop OAM procedures to require that, before there are personnel
reassignments, COs, POs and QAMs perform an inventory of required
documents in their possession or in the official contract file to ensure
maintenance of accurate and complete records.
6.	Update internal procedures and programs to include a process to transfer
all required documents to the newly appointed COs and POs or establish a
key locator system identifying the location of required contracting
documents.
Agency Comments and OIG Evaluations
OARM concurred with recommendations 5 and 6, but the recommendations are
unresolved pending receipt of corrective actions that correspond with the causes
and recommendations.
OARM stated during the exit conference that issues found and addressed under
recommendations 5 and 6 should relate to a management oversight issue, yet we
found no EPA procedures in place to direct the actions of record management
when there are staff reassignments or new appointments. The criteria addressed in
appendix C does not address the local procedures when there are CO, PO and
QAM reassignments and new appointments. In addition, this is a recurring issue
because prior audits recommended improvements in records management that the
agency agreed to yet the conditions still exist (see table 3 above). As a result, we
concluded that this further direction is needed.
13-P-0398
16

-------
Status of Recommendations and
Potential Monetary Benefits
RECOMMENDATIONS
POTENTIAL MONETARY
BENEFITS (In $000s)
Rec.
No.
Page
No.
Subject
Status1
Action Official
Update the policies to include procedures that
would ensure that, for all incentive and time and
material contracts, contract performance metrics
and standards include cost outcomes.
Planned
Completion
Date
Claimed
Amount
Ag reed-To
Amount
Assistant Administrator,
Office of Administration and
Resources Management
12 Update internal programs that provide oversight
and accountability to ensure that, for all incentive
type and time and material contracts, the EPA
performance metrics and standards link to cost
outcomes.
Assistant Administrator,
Office of Administration and
Resources Management
12 Require OTOP, in conjunction with OAM, to review
contractor performance reports generated per the
QASP. Require OTOP to coordinate all actions
taken with the COs and POs and require the COs
to input contractor performance information into
CPARS.
Assistant Administrator,
Office of Administration and
Resources Management
12 Require COs to input interim reports into CPARS
prior to their reassignments.
16 Develop OAM procedures to require that, before
there are personnel reassignments, COs, POs and
QAMs perform an inventory of required documents
in their possession or in the official contract file to
ensure maintenance of accurate and complete
records.
Assistant Administrator,
Office of Administration and
Resources Management
Assistant Administrator,
Office of Administration and
Resources Management
16 Update internal procedures and programs to
include a process to transfer all required
documents to the newly appointed COs and POs or
establish a key locator system identifying the
location of required contracting documents.
Assistant Administrator,
Office of Administration and
Resources Management
0 = recommendation is open with agreed-to corrective actions pending
C = recommendation is closed with all agreed-to actions completed
U = recommendation is unresolved with resolution efforts in progress
13-P-0398	17

-------
Appendix A
Contract Performance Requirements Summary
(Performance Metrics and Standards)
Applicable





performance work





statement


Performance

Frequency of
sections/section #
ID
Metric
standard
Surveillance method
surveillance

1
Percentage of all requests
90%
Monitoring monthly data
As needed


responded to and routed within





15 minutes of submission to the





Service Request Interface during





core hours




2
Percentage of all requests
99%
Monitoring monthly data
As needed


responded to and routed within





60 minutes of submission to the





SRI during core hours



Service request
3
Percentage of tickets correctly
75%
Monitoring monthly data
Monthly
interface

routed or escalated on initial

and Customer Support

(Section 2.1.1)

attempt

Representative input


4
Percentage of tickets correctly
95% (months
Monitoring monthly data
Monthly


routed or escalated on second
05-28)
and Customer Support



attempt
98% (months
Representative input




29-100)



5
Percentage of end-user requests
98%
Monitoring monthly
As needed


(requiring more than 1 business

sample



day to resolve) receiving daily





updates




6
Percentage of assets accurately
98%
Planned sample: cross
Periodic


identified in tracking system

reference random sample





of assets with tracking





system; cross reference





random sample of





tracking system with





assets


7
Percentage of unaccounted assets
99%
Planned sample: audit
Periodic


accurately accounted for and

sampling of missing



recorded in tracking system or

assets records with

Asset management

identified as missing within 30

information in the


days of previous surveillance

Remedy system and

(Section 2.1.2)

check

Ebusiness


8
Percentage of accurate updates
90%
Evaluate moves, adds,
Monthly


made in tracking system within

and changes cross-



1 business day of change being

referenced against date



made

of update made to the





asset tracking system


9
Percentage of accurate updates
99%
Evaluate moves, adds
Monthly


made in tracking system within

and changes cross-



2 business days of change being

referenced against date



made

of update made to the





asset tracking system


10
Percentage of accounts in
90%
Monitoring of monthly
Monthly


compliance with the EPA policy

data provided by Service



and standards

Provider

Account access





and management



Monitor monthly Bind

(Section 2.1.2.3)



view reports


11
Percentage of accounts
99%
Monitoring of monthly
Monthly


established or routed to proper

data provided by Service



authority within 4 hours

Provider

13-P-0398	18

-------
Applicable
performance work
statement
sections/section #
ID
Metric
Performance
standard
Surveillance method
Frequency of
surveillance

12
Percentage of systems that are
patched and have latest virus
definition files by CSIRC required
date
90%
Monthly inspection of
data
Monthly

13
Percentage of systems that are
patched have latest virus definition
files within 10 business days of
CSIRC required date
99%
Monthly inspection of
data
Monthly

14
Percentage of incidents that are
reported to ISO and contracting
officer technical representative
within 4 hours of incident
90%
Random sampling
Monthly
Security
(Section 2.1.3)
15
Percentage of incidents that are
reported to ISO, CSIRC and
COTR within 2 days of incident
99%
Random sampling
Periodic

16
Percentage of incidents of
personally identifiable information
loss that are reported to ISO,
CSIRC and COTR within 1 hour of
incident
90%
Random sampling
Periodic

17
Percentage of incidents of
personally identifiable information
loss that are reported to ISO,
CSIRC and COTR within 4 hours
of incident
99%
Random sampling
Periodic

18
Percentage of contractor staff who
complete required training on time
95%
Random Monitoring
Once a year

19
Percentage of hardware installs
scheduled within 4 hours of
submission of approved request to
the SRI for standard
desktops/laptops or upon receipt
of other hardware
80%
Review monthly data
from SP and validation
through analysis of data
from various sources
Monthly
Hardware
provisioning
(Section 2.2.1.1)
20
Percentage of hardware installs
scheduled within 2 business days
of submission of approved request
to the SRI for standard desktops/
laptops or upon receipt of other
hardware
99%
Review monthly data
from SP and validation
through analysis of data
from various sources
Monthly

21
Percentage of hardware installed
by the scheduled install date and
time; includes loaner hardware
90%
Review monthly data
from SP and validation
through analysis of data
from various sources
Monthly

22
Percentage of hardware installed
within 1 business day of schedule
date and time; includes loaner
99%
Review monthly data
from SP and validation
through analysis of data
Monthly
hardware	from various sources
13-P-0398
19

-------
Applicable
performance work
statement
sections/section #
ID
Metric
Performance
standard
Surveillance method
Frequency of
surveillance

23
Percentage of software installs
scheduled within 4 hours of
submission of approved request to
the SRI for standard software or
upon receipt of other software
80%
Review monthly data
from SP and validation
through analysis of data
from various sources
Monthly
Software
provisioning
(Section 2.2.1.2)
24
25
Percentage of software installs
scheduled within 2 business days
of submission of approved request
to the SRI for standard software or
upon receipt of other software
Percentage of software installed
by the scheduled install date and
time
99%
90%
Review monthly data
from SP and validation
through analysis of data
from various sources
Review monthly data
from SP and validation
through analysis of data
from various sources
Monthly
Monthly

26
Percentage of software installed
within 1 business day of schedule
date and time
99%
Review monthly data
from SP and validation
through analysis of data
from various sources
Monthly

27
Percentage hardware collected
and removed from desk within 1
business day of submission of
approved request to the SRI
90%
Review monthly data
from SP and validation
through analysis of data
from various sources
Monthly
Deprovisioning
(Section 2.3)
28
Percentage hardware collected
and removed from desk within 2
business days of submission of
approved request to the SRI
99%
Review monthly data
from SP and validation
through analysis of data
from various sources
Monthly

29
Percentage of accounts
deactivated/disabled on scheduled
date
99%
Review monthly data
from SP and validation
through analysis of data
from various sources
Monthly
Operations
(Section 2.4)
30
31
Percentage of systems in
compliance with the EPA policy
and standards
Percentage of network printers
fully operational
95%(months
05-16)
98%(months
17-100)
98%
Compare sample of
selected CTS desktop
configurations to the EPA
standard configuration to
ensure compliance
Sampling of printer data
Monthly
Monthly

32
Percentage of preventative
maintenance schedules met
90%
Inspection of reports
Periodic

33
Percentage of password resets
completed within 15 minutes after
notification of request
90%
Sampling monthly data
from remedy
Periodic

34
Percentage of password resets
completed within 60 minutes after
notification of request
99%
Sampling monthly data
from remedy
Periodic
User support
(Section 2.4.3)
35
Percentage of hardware and
software issues resolved or
reprovisioned within 4 hours of
submission of request for service
to the SRI
90%
Review monthly data
from SP and validation
through analysis of data
from various sources
Monthly

36
Percentage of hardware and
software issues resolved or
reprovisioned within 2 business
days of submission of request for
service to the SRI
99%
Review monthly data
from SP and validation
through analysis of data
from various sources
Monthly
13-P-0398
20

-------
Applicable





performance work





statement


Performance

Frequency of
sections/section #
ID
Metric
standard
Surveillance method
surveillance

37
Percentage of moves, changes,
and upgrades completed within
90%
Review monthly data
from SP and validation
Monthly
Moves, changes,

4 hours of scheduled time

through analysis of data




from various sources

and upgrades





(Section 2.4.3.3)
38
Percentage of moves, changes,
99%
Review monthly data
Monthly

and upgrades completed within
2 business days of scheduled time

from SP and validation
through analysis of data
from various sources


39
Percentage of reports delivered in
accordance with reporting
90%
Monitoring submission
dates and quality of
Monthly
Reporting

schedule

reports

(Section 3.5)
40
Percentage of reports delivered
within 5 business days of due date
99%
Monitoring submission
dates and quality of
reports
Monthly

41
Percentage of customer
satisfaction surveys rated at an
average overall score of 4.0 or
higher (scale of 0 - 5 with 5 being
highest)
95%
Review sampling of
surveys
Monthly

42
Percentage of IT training class
evaluations rated at an average
95%
Review sampling of
surveys
Monthly
Customer
satisfaction

overall score of 4.0 or higher




(scale of 0 - 5 with 5 being highest)




43
Percentage of requests properly
completed at first attempt
95%
Monitoring of monthly
data sample and
Customer Support
Representative input
Monthly

44
Percentage of requests properly
completed at second attempt
99%
Monitoring of monthly
data sample and
Customer Support
Representative input
Monthly
* Note: All 44 metrics relate to quality; Metrics 1,2,5, 7-9, 11,13-29, 32-40 relate to timeliness; and Metrics 41-44
relate to business relations.
Source: CTS Contract Modification 6.
13-P-0398
21

-------
Appendix B
PO, CO and QAM Reassignments on CTS Contract

A
IV
J
J
A
S
O
N
D
J
F
IV
A
IV
j
j
A
S
0
N
D
J
F
IV
A
M
J
j
A
S
O
N
D
J
F
IV
A
IV
J
J
A
s
0
N
D
J
F
IV
A
IV
J
J
A
S
p
0
1
5/1/08 -10/8/08
2
10/8/08 - 3/11/09
3
3/11/09 - 12/23/09
4
12/23/09 - 6/21/11
1*
6/21/11 - 9/30/12

A
IV
J
J
A
S
O
N
D
J
F
IV
A
IV
j
j
A
s
0
N
D
J
F
IV
A
M
j
j
A
S
O
N
D
J
F
IV
A
IV
J
J
A
s
0
N
D
J
F
IV
A
IV
J
J
A
S
c
0
1
4/1/08 - 3/11/09
2
3/11/09 - 6/2/10
3
6/2/10 -11/14/11
4
11/14/11 - 9/30/12

A
IV
J
J
A
S
O
N
D
J
F
IV
A
IV
j
j
A
s
0
N
D
J
F
IV
A
M
j
j
A
S
O
N
D
J
F
IV
A
IV
J
J
A
s
0
N
D
J
F
IV
A
IV
J
J
A
S
Q
A
M

vacant
1
9/1/08 -
3/31/11

2
7/3/11 -10/31/11
vacant
3
4/8/12 -
8/5/12


2008
2009
2010
2011
2012
*This CO is the same person but serviced during a different period of time.
Source: OIG analysis.
13-P-0398
22

-------
Appendix C
Relevant Recordkeeping Criteria
The following table illustrates the criteria for government contract records.
Regulation/policy/
instruction/guidance
Requirement/suggestion/recommendation
FAR Subsection 46.104 (c)
Maintain, as part of the performance records of the contract, suitable
records reflecting (1) the nature of government contract quality
assurance actions, including, when appropriate, the number of
observations made and the number and type of defects; and (2)
decisions regarding the acceptability of the products, the processes, and
the requirements, as well as action to correct defects.
EPAAR Subsection
1542.1503 (f)
Copies of the evaluation, contractor response, and review comments
(if any) shall be retained as part of the evaluation, and hard copies shall
be contained in contract files.
EPA Classification No. CIO
2155, EPA Records
Management Policy,
paragraph 8(h)(2)(i)1&3
"All agency employees are responsible for (1) creating and managing
the records necessary to document the agency's official activities and
actions, including those records generated by the EPA contractors and
grantees, in accordance with the EPA recordkeeping requirements;....
(3) filing records for safe storage and efficient retrieval and maintaining
personal papers and non-record materials separately from official
agency records."
EPA Records Management
Manual, Chapter 3
After creating or receiving a record, the EPA staff must capture it by
filing, storing or otherwise systematically maintaining it in a
recordkeeping system. The EPA offices must capture records in a
recordkeeping system that facilitates maintenance and use of the
records in an efficient and cost-effective manner.
EPA CMM, paragraph
7.3.5.5(E)
Contract files must document the contractor's performance to assist the
agency in future procurements.
EPA CMM, Section 11.2,
NOTE (11)
A properly documented file should be kept by both the PO/Delivery
Order PO and work assignment manager, along with copies of invoices
reviewed and approval documents. Relevant work assignment manager
issues should also be communicated and filed in the PO file. The CO
should file formal correspondence in the contract file.
EPA CMM, Section 11.2,
NOTE (15)
The Research Triangle Park Finance Center maintains the official file
documenting invoice payments. The PO/Delivery Order PO and CO
maintain secondary files which generally include added data supporting
payment decisions, suspensions and disallowances.
EPA CMM, Section 11.2,
paragraph 11.2.5.2 (E) -
Filing/Recordkeeping
(1)	POs should update their financial records to reflect information
concerning the invoice, the amount paid, and the account used; and
(2)	both the PO and work assignment manager should maintain files of
approved invoices and all associated documentation. These files will
eventually be sent to the CO at the completion of the contract. The PO
should consolidate PO/work assignment manager invoice files before
they are sent to the CO.
13-P-0398
23

-------
Regulation/policy/
instruction/guidance
Requirement/suggestion/recommendation
EPA CMM, Appendix 42.1,
(4) COR Work plan (a)
Set-up a file system containing all relevant documentation including the
basic contract, list of contracting officer representatives under the
contract, all correspondence and meetings related to the contract,
technical direction, contract deliverables received and reviewed,
payment file and other items that will provide an audit trail of the
contract-level contracting officer representative's actions under the
contract. Maintain files in accordance with agency National Records
Management Program policy. Guidance on maintaining the EPA Series
202, Contract Management Records, is available on the Intranet at
httD://www.eDa.aov/records/Dolicv/schedule/sched/202.htm.
EPA Record Series,
Schedule 202, NARA
Disposition No. N1 -412-06-
6/5, Contract Management
Records
COs and contracting officer representatives should work together to
determine who is responsible for maintaining specific documents to
minimize duplication while still providing an adequate audit trail.
Source: OIG analysis of EPA data.
13-P-0398
24

-------
Appendix D
Agency Comments on Draft Report
#%	mm STATU EflfliQIIlWlL PROTECTION AGENCY
WASHINGTON DC 20*69
*.ฆ
.;'i ฆ 	
, ...".ni .
* H SB
the Customer Tmbvioig/ Selulimm Cmlmi
i--s
ซ*'• v.-P 'VvflfMltt|
OifaoflitlKirititfli!
The Office of Administration and Resources Management and the Office of Environmental
Information have reviewed the draft OIG audit report OA-FY12-0494 and provide the following
comments and corrective actions in response to subject audit findings and recommendations.
13-P-0398
25

-------
General Comments on Report
Page 2 - The OIG writes, "In April 2008, the EPA awarded an indefinite delivery/indefinite
quantity hybrid contract. This hybrid contract also contained provisions for performance-based
incentive fees and fixed hourly rates and material costs for a time and materials contract. This
contract required the contractor to provide technology services to approximately 12,000 program
office users scattered throughout 16 states and the District of Columbia."
Contract EP-W-08-034 was a "Seat Service" contract which refers to all the desktop hardware
and software required for EPA employees to perform work responsibilities. In the contract, the
EPA estimated the maximum number of seats that may be required by the agency was 12,000,
and the contractor proposed a monthly fixed price of $169.75 per seat to ensure all EPA
employees - up to 12,000 - were provided with desktop hardware and software. Ninety percent
of contract spend used this pricing methodology - the Government both ordered and agreed to
pay a fixed amount per seat per month, thus accepting the pricing risk in the event the estimated
12,000 seat calculation was understated, and the contractor accepted the pricing risk in the event
the 12,000 seat calculation was overstated. The remaining 10% of the contract price was for
labor categories proposed by the contractor to support various potential contract tasks, such as
Disaster Recovery and Emergency Response Support and Specialized Projects relating to the
PWS, and other ancillary services that the EPA required in support of seat service. These
additional services were procured via task orders, under which the EPA and the contractor
negotiated the hours needed to perform the required work using the labor categories and fixed
rates set forth in contract. For this effort, the hours were negotiated but the Government agreed to
pay the fixed hourly rates. Given the above-described contract terms and conditions, the OIG's
characterization of the contract as "provide technology services to approximately 12,000
program office users" is a misleading description of a contracting and pricing arrangement.
OIG Evaluation. This report included statements obtained from and supported by the audited
contract. EPA awarded an Indefinite Delivery/ Indefinite Quantity contract that issued task orders for
supplies and services with time and materials provisions that the fixed rates include wages, overhead,
general and administrative expenses and profit. According to the FAR Subsection 16.600, time and
materials contracts are not fixed price contracts and Subsection 16.601 states fixed hourly rates are
supposed to be specified. Based on our review of this contract, modifications (mods) indicated
changes to the rates, hours and user requirements, so the possibility that fixed rates were used and
the overall cost would not change appear to be misleading. Below shows changes to rates per seat:
Year 1	Year 2	Year 3	Year 4
Contract	$169.75	$139.27	$139.43	$140.13
Task Order 2 $175.00 (mod 4) $147.28 (mod 8) $146.70 (mod 15) $147.28 (mod 23).
If the overall cost is determined based on the changing fixed rate multiplied by the seats, the costs will
reflect upward and downward revisions to the stated overall contract price. With these changes, this
contract is not a firm fixed price contract and cost controls are needed. FAR Subsectionl6.601 states
that for a time and materials contract, appropriate government surveillance of contractor performance
is required to give reasonable assurance that efficient methods and effective cost controls are being
used.
13-P-0398
26

-------
Page 8: The OIG writes "By not linking award fee to cost outcomes as required, the EPA did not
effectively hold the contractor accountable for cost inefficiencies and inaccuracies.. .Even though
the agency did not award the contractor an incentive fee, there could have been additional
corrective actions implemented by having cost outcomes tied to contracting activity." Pursuant to
the contract terms and conditions, contractor performance was evaluated against contract
performance metrics, and the contractor was held accountable when performance against those
metrics did not meet the level required for payment of incentive fees. Under the afore-mentioned
contract terms and conditions, performance was incentivized and the contractor was not awarded
an incentive fee when performance did not meet contract metrics. Accordingly, contract
incentives worked exactly as designed and intended in accordance with contract terms and
conditions. Under a fixed price arrangement, controlling costs is inherent in this contract type
and is therefore, not incentivized. The contractor has to deliver the services at the fixed rate in
the contract or absorb any additional costs associated with the services.
OIG Evaluation. According to the FAR Subsection 16.202, a firm fixed price contract provides for a
price that is not subject to any adjustment on the basis of the contractor's cost experience in
performing the contract. A firm fixed price contract type places upon the contractor maximum
incentive for the contractor to control costs and can use award fee or incentives based on factors
other than costs. There is nothing in the audited contract that states this is a fixed price contract. EPA
awarded an Indefinite Delivery/ Indefinite Quantity contract that issued task orders for supplies and
services with time and materials provisions that the fixed rates include wages, overhead, general,
and administrative expenses, and profit. FAR Subsection 16.601 states that a time and materials
contract provides no positive incentive to the contractor for cost control or labor efficiency. Therefore,
appropriate government surveillance of contractor performance is required to give reasonable
assurance that efficient methods and effective cost controls are being used. Since this contract has
an award fee plan, FAR Subsection 16.401 9(e)(2) states award shall not be earned if the overall
cost, schedule and technical performance is not at a satisfactory level.
Responses to the Recommendations
Recommendation 1: Update the policies and procedures to include controls that would ensure
contract performance metrics and standards include cost outcomes for all incentive and time and
materials contracts.
Recommendation 2: Update internal programs that provide oversight and accountability to
ensure that, for all incentive type and time and materials contract, the EPA performance metrics
link to cost outcomes.
QAM Response:
OAM disagrees with these recommendations and disagrees with the OIG's recommendation that
cost is an appropriate incentive for all incentive and time and materials contracts. EPA's CTS
Contract EP-W-08-034 is an Indefinite Delivery/Indefinite Quantity type contract with time and
materials pricing, under which the contractor may earn an incentive fee for performance that
exceeds metrics set forth in the contract.
Attachment 11 of the contract contains contract pricing. Under the Attachment 11 Base Period,
90% of the contract price is for standard seat service, "Seat Service" refers to all the desktop
13-P-0398
27

-------
hardware and software required for EPA employees to perform work responsibilities. In the
contract, EPA estimated the maximum number of seats that may be required by the agency was
12,000, and the contractor proposed a monthly fixed price of $169.75 to ensure all EPA
employees - up to 12,000 - are provided with desktop hardware and software. Again 90% of
contract pricing used this methodology. The other pricing under Attachment 11 Base Period,
which totals 10% of the contract price, is for labor categories proposed by the contractor to
support various potential contract tasks, such as Disaster Recovery and Emergency Response
Support and Specialized Projects relating to the PWS, that the EPA may require in support of
seat service. These additional services were procured via task orders, under which EPA and the
contractor negotiated the hours needed to perform the required work using the labor categories
and fixed rates set forth in Attachment 11. For this effort, the hours were negotiated but the rates
were fixed.
Attachment 6 of the contract sets forth performance standards/metrics for seat service and other
tasks against which contractor performance is measured for the purpose of determining the
award fee. The performance standards/metrics identified are specifically tied to Performance
Work Statement tasks the contractor is performing to ensure all EPA employees have the
necessary desktop hardware and software to perform their work responsibilities. As such, the
criteria set forth in Attachment 6 of the contract is designed to measure critical contract
performance - timeliness, responsiveness, system security, user support, and customer
satisfaction.
Attachment 12 of the contract prescribes how the contractor will be paid an incentive fee for
elevated levels of performance, for example, if the performance metric is met at 100%, the
vendor receives 100% of the incentive fee, if met at 75%, the vendor receives 75%, etc. The final
fee is calculated against the Attachment 6 metrics, and paid against the Attachment 11 pricing.
OAM believes that writing a policy that "includes cost outcomes for all incentive and time and
materials contracts" is not necessary. Additionally, FAR Subpart 16.1 states selecting the
contract type is a matter for negotiation and requires the exercise of sound judgement. Factors
that should be considered in this negotiation are risk, complexity of the requirement, expected
price competition, urgency of the requirement, period of performance of the requirement (the
longer the more risk), and the contractor's technical capability and financial responsibility. The
FAR also states that the objective is to negotiate a contract type and price that will result in
reasonable contractor risk and provide the contractor with the greatest incentive for efficient and
economical performance. In the above pricing arrangement, the number of seats was controlled
by the Government, and the price per seat was fixed, so the only risk to the Government was
with contractor performance. FAR Subsection 16.402 states that performance incentives may be
considered in connection with specific product characteristics (e.g., a missile range, an aircraft
speed, an engine thrust, or vehicle maneuverability) or other specific elements of the contractor's
performance. These incentives should be designed to relate profit or fee to results achieved by
the contractor, compared with specified targets. OAM incentivized contractor performance
characteristics in the areas of timeliness, responsiveness, system security, user support, and
customer satisfaction, and fee was awarded based upon the contractor's performance against
metrics set forth in these areas. Regardless of the amount of fee awarded against these metrics,
the vendor was entitled to collect the fixed price per seat set forth in the contract. Under the
13-P-0398
28

-------
above pricing structure, OAM properly assessed contract risk and established incentive fee
metrics to mitigate and shift the burden of that risk to the contractor. OAM does not agree with
writing a policy requiring cost be an incentive for all incentive and time and materials contracts.
OIG Evaluation. The OAM response does not sufficiently address recommendations 1 and 2.
OAM states OIG is recommending cost as an appropriate incentive, yet we are emphasizing FAR
Subsections 16.401, 16.402, and 16.601, and OMB requirements that, for incentive (contracts with an
award fee) and time and material contracts, agencies link cost outcomes to performance standards
and metrics. Also, OAM's response is stating they have the right to determine contract type when the
issue is with the FAR additional requirements for the contract types already selected. To ensure the
EPA is complying with FAR and OMB requirements addressed above, the EPA policies/procedures
should require that for incentive (contracts with an award fee) and time and material contracts
performance standards and metrics are linked to cost outcomes. In addition for recommendation 2,
per OMB Circular A 123, agencies are responsible for establishing internal controls (policies and
procedures) to achieve compliance with applicable laws and regulations and should design
management structures that help ensure accountability for results.
Recommendation 3: Require OTOP, in conjunction with OAM, to review contractor
performance reports generated per the QASP. Require OTOP to coordinate all actions taken with
the COs and PO's and require the CO's to input contractor performance information into
CPARS.
Recommendation 4: Require CO's to input interim reports into CPARS prior to their
reassignments.
OAM Response:
OAM agrees with these recommendations. With regard to the CTS contract, OAM has been
working with the program office to complete the required past performance evaluation(s). The
contract was registered and interim reports initiated in CPARS on Feb 1, 2013. Evaluation
narratives were drafted by OAM based on information obtained from the award-fee assessments;
and finalization of these reports is pending upon additional input from the program office (OEI).
With regard to overall past performance reporting, OAM believes adequate policy already exists
in the FAR and EPAAR with regard to reporting requirements. As part of the OAM Balanced
Scorecard Performance Measurement and Management Peer Review and Oversight Process,
OAM reviews individual office's procedures for monitoring Contractor Performance Assessment
Reporting System and evaluates whether the office has an effective oversight program.
Furthermore, in March 2013, OFPP began monitoring CPARS compliance and required all
federal agencies to establish compliance goals to be monitored both internally and externally. In
order to monitor and oversee both internal and external past reporting initiatives, OAM
designated a CPARS lead to monitor agencywide past performance reporting compliance on a
monthly basis. The lead ensures compliance by comparing PPIRS reports with the CPARS
database and disseminates the results to all agency CPARS focal points and their managers.
Beginning in February 2013, the CPARS representative is requiring Agency focal points report
status on any incomplete CPARS. Under the above described internal OAM compliance
program, OAM has already improved compliance by 15% and anticipates continual
13-P-0398
29

-------
improvement over the next few months. Additionally, as of June 2013, OAM appears to be on
target to meet our Agency CPARS compliance goal of 75 percent by the end of FY 2013.
Based upon EPA/OAM's established, aggressive and continuous CPARS compliance oversight
and monitoring toward achieving and reporting against external goals, and given CPARS
monitoring is a recognized ongoing effort satisfies recommendation 4 and OARM considers this
recommendation closed.
OIG Evaluation. Even though the agency agreed to recommendations 3 and 4, the comments are
not completely responsive to the recommendations. The comments should address requiring that
COs put interim reports into CPARS prior to their reassignments. This additional procedure will assist
the agency to ensure that the contractor performance reports are entered into CPARS. The response
did not address how or when OARM would implement the agreed-to recommendation to require COs
input interim evaluation prior to reassignments in CPARS. This should be addressed as part of the
60-day response to the final report. According to EPA Manual 2750, if the OIG determines that the
agency's intended corrective actions meet the intent of the recommendations, the OIG reports
recommendations as resolved in the OIG's audit tracking system.
Recommendation 5: Develop OAM procedures to require that before there are personnel
reassignments, CO's, PO's, and QAM's perform an inventory of required documents in their
possession or in the official contract file to ensure maintenance of accurate and complete records.
Recommendation 6: Update internal procedures and programs to include a process to transfer
all required documents to the newly appointed CO's and PO's or establish a key locator system
indentifying the location of required contracting documents.
OAM Response:
OAM agrees with these recommendations although regarding the need for policy updates with
regard to adequate file documentation, OAM believes sufficient internal and external policy and
procedures with regard to contract documentation requirements already exists in the FAR, the
Contract Management Manual, and the Acquisition Handbook. OAM also believes that
meaningful oversight and review processes are necessary to ensure compliance with the
applicable statutes, regulations, policies and procedures. Accordingly, OAM's Balanced
Scorecard Performance Measurement and Management Program consists of self-assessment and
peer review oversight components that utilize a significant file review component to assess
compliance with statues, regulations, policies, and procedures as documented in the contract.
Under this approach, contract file documentation adequacy is internally reviewed by each
division, and also independently by an independent team of senior level contracting
professionals.
Since program implementation in FY 2012, OAM has been assessing the quality and compliance
of EPA's acquisition system agency-wide, and using review results to identify acquisition system
weaknesses, and correct problems areas through training and/or implementation of policy and
process guidance. In peer reviews to date, the need for improved contract file documentation has
been identified as a systems issue finding, and as a result, individual organizations have
developed corrective action plans and OAM has been including documentation requirements
under various comprehensive new acquisition policies on Acquisition Planning and the CPARS.
13-P-0398
30

-------
In addition, Acquisition Handbook, Section 4.1, is being updated to prescribe requirements for
streamlined internal controls (e.g., higher-level transactional reviews), to ensure the sufficiency
and quality of file documentation in support of support contracting officer decisions and actions
and an updated standardized checklist for contract file documentation. Under the BSC PMMP
program, all OAM staff are accountable for the quality and health of the EPA acquisition system.
Information on OAM's BSC PMMP Program may be found on the OAM intranet under the
"Balanced Scorecard" tab on the left.
Based upon EPA/OAM's established and aggressive BSC PMMP compliance oversight and
monitoring program, OAM requests the above described ongoing actions be considered and
these recommendations be closed.
OIG Evaluation. The agency agreed with the recommendations 5 and 6. However, the agency's
comments are not completely responsive to the recommendations. The recommendations discussed
procedures as opposed to policy and OAM procedures are necessary to provide details on what
should be done to ensure maintenance of accurate and complete contract records. The policies
OAM mentioned do not address procedures needed when there are CO, PO and QAM
reassignments and new appointments. Procedures to perform an inventory of the required contract
documents would assist in ensuring maintenance of accurate and complete records. In addition,
procedures are needed to include a process to transfer all required documents to the newly
appointed COs and POs or establish a key locator system indentifying the location of required
contracting documents. According to EPA Manual 2750, if the OIG determines that the agency's
intended corrective actions meet the intent of the recommendations, the OIG reports
recommendations as resolved in the OIG's audit tracking system.
Should you have any questions regarding the OEI issues, please contact Scott Dockum at (202)
566-1914; any questions regarding OAM's responses can be directed to Rebecca Snipe at (202)
564-8346.
13-P-0398
31

-------
Appendix E
Distribution
Office of the Administrator
Assistant Administrator for Administration and Resources Management
Assistant Administrator for Environmental Information and Chief Information Officer
Agency Follow-Up Official (the CFO)
Agency Follow-Up Coordinator
General Counsel
Associate Administrator for Congressional and Intergovernmental Relations
Associate Administrator for External Affairs and Environmental Education
Director, Office of Acquisition Management, Office of Administration and
Resources Management
Director, Office of Technology Operations and Planning, Office of Environmental Information
Audit Follow-Up Coordinator, Office of Administration and Resources Management
Audit Follow-Up Coordinator, Office of Environmental Information
13-P-0398
32

-------