U.S. ENVIRONMENTAL PROTECTION AGENCY
OFFICE OF INSPECTOR GENERAL
Catalyst for Improving the Environment
Quick Reaction Report
EPA Should Delay Deploying
Its New Acquisition System until
Testing Is Completed
Report No. 09-P-0197
July 20, 2009
-------
Report Contributors: Rudolph M. Brevard
Charles M. Bade
Corey Costango
Wen-Tswan Chen
Sabrena Stewart
Abbreviations
EAS EPA Acquisition System
EPA U.S. Environmental Protection Agency
OAM Office of Acquisition Management
OIG Office of Inspector General
RTM Requirements Traceability Matrix
-------
U.S. Environmental Protection Agency
Office of Inspector General
At a Glance
09-P-0197
July 20, 2009
Catalyst for Improving the Environment
Why We Did This Review
We sought to determine to
what extent the U.S.
Environmental Protection
Agency (EPA) has planned
and executed information
system testing to make
informed decisions about the
release of the EPA
Acquisition System.
Backgrounds
EPA acquires approximately
$1.3 billion in goods and
services annually. The Office
of Acquisition Management
(CAM), within the Office of
Administration and Resources
Management, is responsible
for managing the Agency's
procurement of products and
services. A strategic goal of
OAM is "optimizing business
processes." As a part of this
effort, OAM is developing a
new acquisition system, the
EPA Acquisition System.
For further information,
contact our Office of
Congressional, Public Affairs
and Management at
(202)566-2391.
To view the full report,
click on the following link:
www.epa.qov/oiq/reports/2009/
20090720-09-P-0197.pdf
EPA Should Delay Deploying Its New Acquisition
System until Testing Is Completed
What We Found
OAM did not comply with EPA's System Life Cycle Management policy and
procedure while developing the new EPA Acquisition System (EAS). OAM did not
fully develop the system's requirements documents during the requirements phase and
requirements were incomplete. Test scripts were not developed to prove that the
system fulfilled all requirements and ensure that the system would function as required.
Although the EAS Project Manager developed a Draft Master Test Plan that contained
testing procedures, OAM management never approved, implemented, and enforced this
plan.
OAM management did not provide the oversight, authority, and support necessary to
ensure the EAS development project complied with EPA's System Life Cycle
Management policy and procedure. Because OAM had not completed the steps needed
to reasonably ensure that EAS would meet EPA's business needs if implemented by
June 29, 2009, as planned, OAM does not have a sound basis for deploying EAS as
scheduled. More management emphasis is needed to ensure the system development
control environment achieves the desired results and the end product meets EPA's
needs.
What We Recommend
We recommend that the Assistant Administrator for Administration and Resources
Management:
• Identify and document all system requirements, including functional, technical,
security, and EPA-specific requirements, in the EAS Requirements
Document(s).
• Update, review, and implement formal testing policies and procedures.
• Delay implementing EAS until OAM has successfully tested all system
requirements.
• Update the EAS Project Schedule to communicate the current status of and
future plans for EAS project activities.
• Develop and implement oversight procedures to ensure that further EAS system
development activities and future projects adhere to all requirements.
During a meeting on May 27, 2009, OAM management agreed with our findings and
informed the audit team that OAM has delayed EAS deployment until after the fiscal
year end.
-------
\ UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
0 S WASHINGTON, D.C. 20460
'"
OFFICE OF
INSPECTOR GENERAL
July 20, 2009
MEMORANDUM
SUBJECT: EPA Should Delay Deploying Its New Acquisition System
until Testing Is Completed
Report No. 09-P-0197
FROM: Rudolph M. Brevard
Director, Information Resources Management Assessments
TO: Craig H. Hooks
Acting Assistant Administrator for Administration and Resources Management
This is our Quick Reaction Report on the subject audit conducted by the Office of Inspector
General (OIG) of the U.S. Environmental Protection Agency (EPA). This report conveys
significant time-sensitive issues the OIG has identified and corrective actions the OIG
recommends. This report represents the opinion of the OIG and does not necessarily represent
the final EPA position. Final determinations on matters in this report will be made by EPA
managers in accordance with established audit resolution procedures.
The estimated cost of this report - calculated by multiplying the project's staff days by the
applicable daily full cost billing rates in effect at the time - is $249,019.
Action Required
In accordance with EPA Manual 2750, you are required to provide a written response to this
report within 90 calendar days. You should include a corrective actions plan for agreed upon
actions, including milestone dates. We would like to thank your staff for their cooperation. We
have no objections to the further release of this report to the public. This report will be available
at http://www.epa.gov/oig.
If you or your staff have any questions regarding this report, please contact me at (202) 566-0893
or brevard.rudy@epa.gov: or Charles M. Dade, Project Manager, at (202) 566-2575 or
dade.chuck@epa.gov.
-------
EPA Should Delay Deploying Its New 09-P-0197
Acquisition System until Testing Is Completed
Table of Contents
Purpose 1
Background 1
Scope and Methodology 2
Findings 2
OAM Did Not Fully Develop System Requirements and Test Scripts 3
OAM Did Not Approve, Implement, and Enforce Testing Procedures 4
OAM Lacked Approval Documentation 5
Recommendations 6
Agency Comments and OIG Evaluation 6
Status of Recommendations and Potential Monetary Benefits 8
Appendix
A Distribution 9
-------
09-P-0197
Purpose
We sought to determine to what extent the U.S. Environmental Protection Agency (EPA) has
planned and executed information system testing to make informed decisions about the release of
the EPA Acquisition System (EAS). Properly testing the system prior to deployment provides
management with the information needed to ensure that the system will meet all of EPA's
business needs and comply with federal and EPA requirements.
Background
EPA indicated it acquires approximately $1.3 billion in goods and services annually. These
acquisitions directly support EPA's mission to protect human health and safeguard the
environment. The Office of Acquisition Management (OAM), within the Office of
Administration and Resources Management, manages the Agency's procurement of goods and
services. A strategic goal of OAM is "optimizing business processes." As a part of this effort,
OAM is in the process of developing a new acquisition system - EAS.
According to OAM's EAS Update briefing provided at the Administrative Officers/Management
Analysts Forum on May 6, 2009, initial deployment of EAS was scheduled for June 29, 2009.
EPA indicated EAS would provide a commercial-off-the-shelf application that can be accessed
from any EPA or secure flexiplace site through the Intranet. Buyers, contract specialists, and
contracting officers throughout EPA Headquarters, regions, and laboratories will use the system
to create and manage contracts and purchases. EPA indicated EAS would provide improved
internal and external reporting, and permit performing acquisition and business functions in a
streamlined, secure, and modern manner. EPA disclosed it would integrate EAS with the
Agency's financial system. The Agency indicated EAS would be used from acquisition request
through contract close-out.
EPA's System Life Cycle Management Policy promotes effective and efficient solutions for
designing and operating information systems. Consistent with the policy, EPA's System Life
Cycle Management Procedure requires periodic, documented, management-level review of
projects by the sponsoring office. The procedure requires program managers to oversee System
Life Cycle Management activities and establishes key opportunities to review development as
the project progresses. Major decision points are called "control gates." At each control gate,
the system manager must present the required System Life Cycle Management documentation
corresponding to the phase being reviewed to receive appropriate management approval.
The EPA System Life Cycle Management Procedure dictates that the requirements phase be
successfully completed prior to starting the system acquisition/development phase.
Requirements must be documented and clearly define what the system must do to satisfy the
business need. Further, the test phase must be successfully completed prior to moving into the
implementation phase. The objective of the test phase, during which system tests are conducted
and evaluated, is to prove that the developed system satisfies all requirements.
The Federal Managers' Financial Integrity Act of 1982 requires Federal Agencies to establish
and maintain a reliable internal control structure and evaluate and report on implementation of
-------
09-P-0197
internal controls. The report must include whether EPA financial management systems such as
EAS comply with federal requirements; the EPA Administrator reports instances where systems
do not conform.
The Federal Financial Management Improvement Act of 1996 requires the EPA Administrator to
determine whether EPA's financial management systems substantially comply with the Act's
requirements. If the Administrator determines that systems are not compliant, the Administrator
must establish a remediation plan. EPA must also determine whether these noncompliances
should be reported as nonconformances with Section 4 of the annual Federal Managers'
Financial Integrity Act statement.
Scope and Methodology
We performed this audit from February through May 2009 at EPA Headquarters in Washington,
DC. We conducted the audit in accordance with generally accepted government auditing
standards. These standards require that we plan and perform the audit to obtain sufficient and
appropriate evidence to provide a reasonable basis for our findings and conclusions based on the
audit objectives. We believe the evidence obtained provides a reasonable basis for our findings
and conclusions.
We assessed the EAS testing process, and considered relevant internal controls associated with
the scope of our review. We performed a limited review of relevant System Life Cycle
Management documentation, such as project management plans, system requirements
documents, system testing documents, requirements traceability matrices, and evidence of
management reviews and approvals, based on requirements in applicable EPA policies and
procedures and related federal regulations. We also conducted interviews with the EAS Project
Management Team and contracting staff overseeing contracts associated with this system
project.
We had not performed past audits of EAS. Thus, we performed no follow-up during this audit.
Findings
OAM did not comply with EPA's System Life Cycle Management policy and procedure while
developing the new EAS system. OAM did not fully develop the system's requirements
documents during the requirements phase. The requirements were incomplete and test scripts
were not developed to prove that the system fulfilled all requirements and ensure that the system
would function as required. Although the EAS Project Manager developed a Draft Master Test
Plan that contained testing procedures, OAM management never approved, implemented, and
enforced the plan. OAM management did not provide the oversight, authority, and support
needed to ensure EAS development complied with EPA's System Life Cycle Management
policy and procedure. Because EPA did not reasonably ensure that EAS would meet EPA's
business needs if it had been implemented by June 29, 2009, as planned, OAM does not have a
sound basis for deploying EAS as scheduled.
-------
09-P-0197
EPA plans to interface EAS with the following four systems:
• EPA's Financial Replacement System (EPA's new core financial management system
being developed to replace EPA's Integrated Financial Management System; EAS would
interact with the Integrated Financial Management System until it is replaced)
• Contract Payment System
• Payroll System
• Financial Data Warehouse
Given the above noted weaknesses and the substantial role EAS plays in EPA's financial
reporting, more management emphasis is needed to ensure the system development control
environment achieves the desired results and the end product meets EPA's needs. We believe that
if EAS is deployed as planned, management would not have sufficient basis to determine whether
EAS "substantially complies" with the Federal Financial Management Improvement Act.
Consequently, EPA's Administrator may have to declare a substantial noncompliance in the
Fiscal Year 2009 Financial Statement audit report and declare a material weakness in conjunction
with EPA's 2009 Federal Managers' Financial Integrity Act annual assurance process.
OAM Did Not Fully Develop System Requirements and Test Scripts
The EAS Project Manager stated there is an unknown number of newly defined EAS
system requirements that OAM had not documented in the formal EAS requirements
documents. This occurred because the Program Management Office created more system
requirements during meetings with the EAS system developer and EPA stakeholders but
did not keep track of and update the requirements documents. In response to our request,
the EAS Project Manager tasked the support contractor with going through meeting
minutes and identifying and updating the requirements documents. As of May 6, 2009,
the Project Manager said the process was still taking place. As a result, EAS might be
developed without the missing requirements and system testing may not validate that the
system fulfills the missing requirements and functions as required.
In addition, about one-fourth of the documented EAS system requirements did not have
test scripts to ensure the system successfully meets EPA's needs. This occurred because
OAM did not properly maintain the requirements traceability matrix to ensure that each
requirement mapped to a test script. From July 2007 through April 2009, EAS functional
and technical system requirements grew from approximately 410 to 1,350. In response to
our request for a crosswalk of EAS system requirements and developed testing scripts,
EPA conducted an analysis and indicated approximately 350 requirements were not
mapped to a test script. The importance of capturing all requirements and developing
associated test scripts is noted in OAM's Draft Master Test Plan:
"The RTM [Requirements Traceability Matrix] serves as the basis for all
project requirements management activities, by providing a traceability of
each requirement and ensuring that tests map to all requirements. Every
item in the RTM must first be uniquely identified it is critical to
ensure that all requirements are captured and Test Scripts are developed
-------
09-P-0197
for each requirement. If a requirement is missing from the list, then there
is a significant risk that the requirement will not be tested, resulting in
incomplete testing. As a result, the product risks the possibility of passing
testing without a thorough test of all requirements. "
By not maintaining the requirements traceability matrix, management did not have
reasonable assurance that each of the requirements was captured and tested to ensure the
system would function as required.
OAM Did Not Approve, Implement, and Enforce Testing Procedures
EPA management did not have a reasonable basis to rely upon the current EAS testing
process because OAM had not approved, implemented, and enforced testing procedures.
Further, OAM provided the vendor with too much control over developing the system
testing scripts without independent verification.
The vendor developing EAS also developed the test scripts used by EPA for the
Government Acceptance Testing and User Acceptance Testing. We found no
compensating controls in place to independently validate the completeness and adequacy
of test scripts prior to testing. The Government Acceptance Testing took place between
December 2008 and April 2009, but the test scripts did not go through a formal review
until April 2009 in response to our request for a complete listing of system requirements
and a crosswalk showing each requirement mapped to a test script. Had EPA followed its
Draft EAS Master Test Plan, all of the EAS requirements and test scripts would have
been reviewed and approved at least 2 weeks before the start of testing.
Although OAM hired a second contractor to help with EAS project management, OAM
did not fully use the second contractor to track and reconcile system requirements and
test scripts. OAM had not:
• Implemented a change management procedure to ensure EAS project changes are
vetted with all parties, approved by EPA management, and incorporated into the
EAS system development process.
• Verified that testing scripts provided by the system developer were correctly
designed to test the prescribed system requirements.
• Followed the steps outlined in the Draft EAS Master Test plan that requires EPA
management to verify the test results once they complete a system test.
• Defined specific criteria in the Draft EAS Master Test Plan to indicate what
constitutes a successful system test.
The EAS Project Manager could not provide any documented evidence to show that
anyone reviewed system requirements for accuracy and completeness before executing
-------
09-P-0197
the Government Acceptance Testing. The Project Manager also could not provide
documented evidence of formal review and approval of test results. Even though some
test results reports were generated, there was no evidence that those reports were
reviewed and approved because there was no formal signoff on those reports. If the Draft
EAS Master Test Plan were approved, implemented, and enforced, all of the EAS
requirements and test scripts would have been reviewed and signed off on by the
Functional Lead and Technical Lead before the start of the Government Acceptance
Testing. Further, key reports would have been used to summarize the testing results and
provide insight on the system's success at meeting requirements. In addition, each round
of testing would not have been successfully completed until all requirements passed
testing or were marked for a future release by EPA and the results were formally
approved by EPA.
Without independently verifying the accuracy and completeness of system requirements
and test scripts prior to testing, OAM cannot properly plan and execute its testing.
Without reliable test scripts and documented independent reviews of test results, OAM
will not have the reasonable assurance that the system satisfies defined requirements and
mission needs. Further, inconsistencies in interpreting test results could result from not
having criteria for defining a successful test. As a result, OAM would not have had a
reasonable basis for concluding that EAS was ready for deployment by the June 29, 2009,
planned deployment date.
OAM Lacked Approval Documentation
OAM lacked approval documentation to support that the EAS Project Team completed
all the required system development activities outlined in EPA's System Life Cycle
Management policy and procedure.
Our review focused on EAS testing processes; as such, we did not perform detailed
reviews of all EAS project management documents. However, in several instances,
management could not provide documentation that it met prescribed Agency
requirements. For example, OAM proceeded to the Acquisition/Development Phase,
where it issued the Request for Proposal and made a vendor selection prior to completing
all of the required documents for the requirements phase. OAM had only identified about
30 percent (410 of 1,350) of the requirements as of the time of the Request for Proposal.
In addition, the EAS Project Management Team could not provide evidence that OAM:
• Performed a feasibility study during the EAS system planning phase.
• Approved and implemented the Configuration Management Plan during the
system planning phase.
• Completed and approved the Requirements Documents during the requirements
phase.
-------
09-P-0197
• Obtained the Chief Architect's certification on the EAS Solution Architecture
before the EAS project moved into the development phase.
The absence of key EAS documentation is an indicator that the management control
structure for EAS development requires more emphasis. Project reviews and subsequent
approvals provide management with a reasonable basis to evaluate whether the EAS
Project Management team took the necessary steps to manage the project's risks and
ensure the developed application meets EPA's needs.
Recommendations
We recommend that the Assistant Administrator for Administration and Resources Management:
1. Identify and document all system requirements, including functional, technical, security,
and EPA-specific requirements, in the EAS Requirements Document(s).
2. Update, review, and implement formal testing policies and procedures that would
enforce:
a. the review and approval of all system requirements prior to testing,
b. the completion of Requirement Traceability Matrices mapping each system
requirement to a test script,
c. the independent validation of vendor-supplied test scripts prior to testing, and
d. the review and approval of testing results at the end of each round of testing.
3. Delay EAS implementation until OAM has successfully tested all of the system
requirements.
4. Update the EAS Project Schedule to communicate the current status of and future plans
for EAS project activities.
5. Develop and implement oversight procedures to ensure that further EAS system life cycle
activities, as well as any future System Development/Acquisition projects, adhere to all
requirements outlined in EPA's System Life Cycle Management policy and procedure.
Agency Comments and OIG Evaluation
On May 27, 2009, the audit team met with OAM management to discuss the finding outline for
our report. OAM management agreed with our findings, acknowledged our concerns regarding
prematurely deploying the system, and informed the audit team that OAM has delayed EAS
deployment until some time after the fiscal year end. OAM management contended that using
vendor-supplied testing scripts does not automatically mean the testing process is fundamentally
flawed. OAM management also stated that they have documentation to support test script
review activities.
-------
09-P-0197
We changed the report, where appropriate, to address OAM management's concerns. We agree
that using vendor-supplied testing scripts does not automatically mean the testing process is
fundamentally flawed. However, performing thorough, independent reviews of those vendor-
supplied testing scripts will provide a compensating control for the lack of independence and
significantly improve the process. After our meeting with OAM management, the audit team
received an Excel spreadsheet that contained a list of test scripts and comments. However, based
on the spreadsheet alone, we could not determine who performed the test script review nor
ascertain that OAM management reviewed and approved the entire listing of test scripts.
Therefore, the audit team did not consider it as acceptable evidence of test script review and
approval. We consider all of the recommendations open with agreed-to actions pending.
-------
09-P-0197
Status of Recommendations and
Potential Monetary Benefits
RECOMMENDATIONS
Rec. Page
No. No.
Subject
Status1
Action Official
Planned
Completion
Date
POTENTIAL MONETARY
BENEFITS (in $OOOs)
Claimed Agreed To
Amount Amount
Identify and document all system requirements,
including functional, technical, security, and EPA-
specific requirements, in the EAS Requirements
Document(s).
Update, review, and implement formal testing
policies and procedures that would enforce:
a. the review and approval of all system
requirements prior to testing,
b. the completion of Requirement Traceability
Matrices mapping each system requirement
to a test script,
c. the independent validation of vendor-supplied
test scripts prior to testing, and
d. the review and approval of testing results at
the end of each round of testing.
Delay EAS implementation until 0AM has
successfully tested all of the system requirements.
Update the EAS Project Schedule to communicate
the current status of and future plans for EAS
project activities.
Develop and implement oversight procedures to
ensure that further EAS system life cycle activities,
as well as any future System Development/
Acquisition projects, adhere to all requirements
outlined in EPA's System Life Cycle Management
policy and procedure.
Assistant Administrator for
Administration and
Resources Management
Assistant Administrator for
Administration and
Resources Management
Assistant Administrator for
Administration and
Resources Management
Assistant Administrator for
Administration and
Resources Management
Assistant Administrator for
Administration and
Resources Management
0 = recommendation is open with agreed-to corrective actions pending
C = recommendation is closed with all agreed-to actions completed
U = recommendation is undecided with resolution efforts in progress
-------
09-P-0197
Appendix A
Distribution
Office of the Administrator
Acting Assistant Administrator for Administration and Resources Management
Acting Assistant Administrator for Environmental Information and Chief Information Officer
Acting Chief Financial Officer
Acting Director, Office of Technology Operations and Planning
Agency Follow-up Official (the CFO)
Agency Follow-up Coordinator
Acting General Counsel
Acting Associate Administrator for Congressional and Intergovernmental Relations
Acting Associate Administrator for Public Affairs
Audit Follow-up Coordinator, Office of Administration and Resources Management
Acting Inspector General
------- |