^ED3X £ A \ ISS,' Office of Inspector General Audit Report Information Technology EPA Management of Information Technology Resources Under The Clinger-Cohen Act Report No. 2002-P-00017 September 30, 2002 ------- Inspector General Division Conducting the Audit: Information Technology Audits Division Program Offices Involved: Office of Environmental Information Office of the Chief Financial Officer Office of Enforcement and Compliance Assurance Office of Air and Radiation Office of Solid Waste and Emergency Response Office of Water Audit Team Members: Jim Rothwell, Project Manager Jim Haller, Technical Support Ernest Ragland, Auditor Michael Young, Auditor Robert Shields, Auditor Robert Smith, Auditor Abbreviations CIO Chief Information Officer CPIC Capital Planning and Investment Control CTO Chief Technology Officer DCIOT Deputy CIO for Technology EPA U.S. Environmental Protection Agency GAO General Accounting Office ICIS Integrated Compliance Information System IRM Information Resources Management IIS Information Investment Subcommittee IT Information Technology I-TIPS Information Technology Investment Portfolio System OEI Office of Environmental Information OIG Office of Inspector General OMB Office of Management and Budget RCRAInfo Resource Conservation and Recovery Act Information Management System SDWIS/STATE Safe Drinking Water Information System/State Version Modernization Effort SMP System Management Plan VPN Virtual Private Network ------- UNITED STATES ENVIRONMENTAL PROTECTION AGENCY WASHINGTON, D C. 20460 15SU) PRO^° SEP 30 2002 OFFICE OF THE INSPECTOR GENERAL MEMORANDUM SUBJECT: FROM: TO: Final Report: EPA's Management of Information Technology Resources under the Clinger-Cohen Act Audit No. 2001-0591 Report No. 2002-P-00017 Patricia H. Hi Director, Business Systems (2421Tj Kim Nelson Assistant Administrator and Chief Information Officer (281 OA) Attached is our report titled "EPA's Management of Information Technology Resources under the Clinger-Cohen Act." Our objective was to evaluate whether EPA has established a Chief Information Officer (CIO) position with sufficient authority and administrative controls to effectively manage Information Technology (IT) resources agency-wide, and to assess whether the CIO has adequately implemented the Act's requirements. The audit also evaluated whether the CIO coordinated with the Chief Financial Officer to help provide sufficient direction and guidance to Agency managers to ensure IT investments are acquired in a cost-effective manner. This audit report contains findings that describe pioblems the Office of Inspector General (OIG) has identified and corrective actions the OIG recommends. This audit report represents the opinion of the OIG, and the findings contained in this audit report do not necessarily represent the final EPA position. Final determinations on the matters in the audit report will be made by EPA managers in accordance with established EPA audit resolution procedures. Action Required In accordance with EPA Order 2750, you, as the primary action official, are required to provide us with a written response to the audit report within 90 days of the final report date. If corrective actions will not be complete by the response date, we ask that you describe the actions that are ongoing and reference specific milestone dates which will assist us in deciding whether to close this report. In addition, please track all action plans and milestone dates in EPA's Management Audit Tracking System. Recycled/Recyclable .Printed with Vegetable Oil Based Inks on 100% Recycled Paper (40% Postconsumer) ------- 2 We appreciate the cooperation afforded us during the course of this audit by the Office of Environmental Information, Office of Chief Financial Officer, Office of Air Quality Planning and Standards, Office of Water, and Office of Solid Waste and Emergency Response. We have no objections to the further release of this report to the public. Should you or your staff have any questions regarding this report, please contact James Rothwell, Project Manager for Information Technology Audits Division, at (202) 566-2570. Attachment ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Executive Summary Introduction In 1996, the U.S. Congress enacted the Clinger-Cohen Act (Act), initially known as the Information Technology Management Reform Act, to improve the management of federal agencies' information technology (IT) resources. The Act requires each agency head to develop and implement a process for maximizing the value of and assessing and managing the risks of IT acquisitions. This process is known as the IT Capital Planning and Investment Control (CPIC) process. The CPIC process relates to an agency's selection of information technology investments, the management of such investments, and the on-going evaluation of funded investments. The Act requires the Chief Information Officer (CIO) to establish an Enterprise Architecture and to use it as part of the CPIC process. The Enterprise Architecture establishes the entity-wide road map to achieve an agency's mission. An agency's capital planning and control process must build from its current Enterprise Architecture, and support the transition from its current to target architecture. Objectives We audited to determine whether: • EPA has established a CIO position with sufficient authority and administrative controls to effectively manage IT resources Agency-wide. • EPA's CIO has adequately: / managed and controlled investments using a comprehensive IT CPIC process; / developed and maintained an Enterprise Architecture; / monitored IT investment projects and provided standard tools and practices for managing system development projects; and / coordinated with the Chief Financial Officer to help provide sufficient direction and guidance to Agency management regarding cost effective acquisitions. Results in Brief EPA's CIO has sufficient authority to shape and direct Information Resource Management (IRM) activities. Nevertheless, past CIOs have not provided the leadership needed to fully implement the changes required by the Act. Since established in 1998, EPA's CIOs have taken some actions to implement and institutionalize the Agency-wide authority and responsibilities for IT capital investments. Yet many strategic planning and development activities only started in 1 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act fiscal 2001. A first step in addressing EPA's planning needs was the CIO approval of an updated EPA Strategic Information Plan on July 29, 2002. EPA's new CIO recognizes the importance of the issues raised in this report and is taking aggressive steps to address the Act's fundamental components. For example, in May 2002, the CIO established a Chief Technology Officer position to coordinate, implement, and advise on the Strategic Technology Plan, Agency Architecture, E-government activities and IT investments. Also, in June 2002, the Deputy CIO for Technology (DCIOT) was assigned responsibility for establishing and publishing standards and procedures based on the Act. However, institutionalizing structured, centralized controls and oversight processes will take additional effort and resources. Some program managers have not taken the Act seriously and have viewed its requirements as another step to satisfy the annual OMB budget call. Several key factors continued to limit the realization of a successful program: • Senior program managers continued to use outdated and unauthorized IT acquisition practices, because Agency IT policies conflicted with the Act's requirements and the CIO's authority. • The Agency was still developing its Enterprise Architecture Plan, and had not established a formal chain of command, either through policies or formal delegation, from the CIO to the Chief Technology Officer, DCIOT, and Chief Architect. In particular, formalization of the Chief Technology Officer and Chief Architect positions will help ensure sufficient management authority and resources to implement the Act. Also, position descriptions for all three roles should be updated to address respective responsibilities for the development of an Enterprise Architecture and execution of related IT activities. • EPA had not implemented a CPIC performance-based measurement system for assessing and managing risks of IT acquisition, and implementing, monitoring and evaluating IT projects. EPA is in the process of implementing an IT cost accounting system to support such areas as IT budget reporting, project management, and system life cycle management. Project cost accounting is a critical management tool for EPA to achieve acceptable, efficient and effective accounting, budgeting, and procurement of IT investment projects. With regard to the fiscal 2002 budget, we believe the CIO had minimal assurance that IT investments reported to OMB would maximize their value. Moreover, the CIO had little assurance that these investments were adequately assessed for risk factors, that risks were being managed, or that products were procured consistent with the Act's requirements. EPA reported investments that totaled more than $449 million for the 11 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act fiscal 2002 budget. Our review showed that EPA continued to spend millions on IT investments that appeared to be making minimal or insignificant progress. During the period under review, EPA's IT investments were not maximizing the efficiency of IT operations nor resolving long-standing problems, such as integration of environmental data. Existing IT contracts, with a maximum value totaling approximately $1.6 billion, can be awarded new work without proper delegated authorization from the CIO. Furthermore, EPA continued to award new IT contracts without required CIO approval. Recommendations Improving the fundamental issues addressed in this report will require a series of inter- related corrective actions. To help EPA management plan for and channel its resources in a methodical manner, we prioritized the recommendations listed in Chapters 2 through 6 of this report. The most prominent recommendations are summarized below. The CIO will need to complete and implement these actions in order to improve the way EPA's IT investments are assessed, managed, and evaluated. • Revise outdated policies to remove unauthorized IT business practices and add new requirements. • Formally re-delegate authority and responsibilities for implementing the Clinger- Cohen Act to the Chief Technology Officer and, in turn, further re-delegate to the Chief Architect the management authority and responsibilities for maintaining an Enterprise Architecture. • Establish and update policies for the Enterprise Architecture and execution of related IT investment activities under the Act. • Implement an automated project management system. • Implement individual project monitoring and evaluation processes for IT investments. The CIO also will need to work with other Agency officials to establish delegations, policies, and procedures for IT procurements. Agency Comments and OIG Evaluation We received comments from EPA's CIO, Comptroller, Assistant Administrator for Solid Waste and Emergency Response, and the Director, Information Transfer and Program Integration Division of the Office of Air and Radiation. We amended the report based on these responses, as well as additional discussions with appropriate management officials. in Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act The CIO agreed with our emphasis on the importance of an effective IT investment management program and agreed to continue to aggressively address issues identified by the report. The CIO noted substantive accomplishments toward that goal, such as establishing new policies, promulgating a new information strategic plan, hiring a Chief Technology Officer, employing a risk-based process for IT investments, and establishing a cost tracking system. While we agree that EPA has taken significant initial steps to address the report's findings and recommendations, there are still significant recommendations that need to be addressed, such as implementing an automated system to manage the CPIC process. Also, authorities and responsibilities for the Chief Technology Officer and Chief Architect need to be incorporated into Agency policy, and resources need to be dedicated to complete and maintain EPA's Enterprise Architecture. The CIO has established an ambitious schedule to address this report's recommendations, and it will require EPA to continue dedicating significant resources. The Comptroller responded that his office was working with an Office of Environmental Information workgroup to ensure consistent treatment of IT costs with common system life cycle stages. The Comptroller did not agree to amend existing IT contracts and stated that the interim policy announcement provided adequate controls. We still have concerns about the adequacy of the new cost accounting process for categorizing project costs by life cycle phases. However, we will defer making formal recommendations until a more detailed assessment of the new process can be competed as part of the Fiscal 2002 financial statements audit. The Assistant Administrator for Solid Waste and Emergency Response, and the Director for the Office of Air and Radiation's Information Transfer and Program Integration Division, both disagreed with our conclusion that project management controls were inadequate. We did not review all project management controls, but we did document inaccurate and/or unsupported information being reported as part of the budget for the IT system projects. We also found that the projects did not comply with existing Agency systems development life cycle policy documentation requirements. We consider these to be significant project management weaknesses. iv Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Table of Contents Executive Summary i Chapters 1 Introduction 1 2 CIO Needs to Fully Implement Clinger-Cohen Act Requirements 5 3 Weaknesses in CPIC Process Place EPA's IT Investments at Risk 11 4 EPA Needs to Organize and Integrate Planning for IT Investments 19 5 EPA Needs to Strengthen IT Project Management Controls 29 6 Project Cost Accounting System Vital for Planning and Managing IT Investments 37 Appendices 1 Details on Scope and Methodology 41 2 Office of Environmental Information's Response to Draft Report 45 3 Office of the Chief Financial Officer's Response to Draft Report 67 4 Office of Air Quality Planning and Standards' Response to Draft Report 71 5 Office of Solid Waste and Emergency Response's Response to Draft Report 73 6 Report Distribution 75 v Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act VI Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Chapter 1 Introduction Purpose The audit's objectives were to determine whether: • EPA had established a Chief Information Officer (CIO) position with sufficient authority and administrative controls to effectively manage Information Technology (IT) resources Agency-wide; • EPA's CIO had adequately: / Managed and controlled investments using an IT Capital Planning and Investment Control (CPIC) process, including a determination of whether investment decisions minimize the risk to the Agency, provide a positive return on investment, and satisfy the Clinger-Cohen Act requirements; / Adopted the Federal Enterprise Architecture Framework components necessary for developing and maintaining an Agency Enterprise Architecture, as prescribed by the Office of Management and Budget (OMB) and the Federal Chief Information Officers Council; / Monitored IT investment projects and provided standard tools and practices for managing system development projects; and / Coordinated with the Chief Financial Officer to help provide sufficient direction and guidance to Agency management to ensure IT investments were acquired in a cost-effective manner. Background and Criteria Act Established CIO Role and CPIC Process The Clinger-Cohen Act of 1996 (Public Law 104-106) intended for a central process, led by a CIO, to manage IT investments across an agency. Since 1996, EPA has taken two significant actions to implement the Clinger-Cohen Act. In 1998, EPA's Administrator established the CIO position through Delegation 1-84. The Delegation assigned responsibility to exercise all responsibilities of the CIO pursuant to the Clinger- Cohen Act, such as establishing an IT Architecture and an IT CPIC process. Then, in 1999, EPA reorganized its Agency IT management, and established an Office of Environmental Information (OEI) and a Quality Information Council. 1 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act The Act requires the CIO to implement a CPIC process for maximizing the value and assessing and managing the risks of an agency's IT acquisitions. The CPIC process is to provide for the selection of investments using minimum criteria, both quantitative and qualitative, for comparing and prioritizing alternative information systems projects. In addition, the CPIC process must provide a means for senior managers to obtain timely information regarding progress (at established milestones). The Act identifies numerous requirements and responsibilities for the agency head, CIO, and other key officials. Specific responsibilities for the CIO include: • Developing and implementing a sound and integrated Enterprise Architecture; • Monitoring and evaluating the performance of IT programs based on defined measurements, and determining whether to continue, modify, or terminate a program or project; • Implementing and enforcing applicable government-wide and Agency IT management policies, principles, standards, and guidelines; • Acquiring and managing information resources in a manner consistent with Federal laws and internal policies and procedures. • Integrating Information Resources Management (IRM) operations and decisions with organizational planning, budget, financial management, and program decisions; • Developing a full and accurate accounting of IT expenditures, related expenses, and results; and • Establishing a process to select, control, and evaluate the results of major information system initiatives. Law and OMB Circulars Further Define Requirements Under Title 44, U.S. Code, Section 3506, agencies are responsible for developing and maintaining an IRM strategic plan, as well as a current and complete inventory of its information resources. OMB Circular A-130, Management of Federal Information Resources, requires the CIO to: (1) prepare and update a cost-benefit analysis for each information system, as necessary throughout its life cycle; (2) conduct cost-benefit analyses to support ongoing management oversight processes; (3) conduct post-implementation reviews of information systems to validate estimated benefits and document effective management practices; and (4) establish information system management oversight mechanisms. This Circular also emphasizes that IRM planning should help the Agency link IT to mission needs. Furthermore, IRM planning should coordinate with other agency 2 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act planning processes, including strategic, human, and financial resources. The agency should employ mechanisms to ensure that major information systems proceed in a timely fashion towards agreed-upon milestones, meet user requirements, and deliver intended benefits to the agency and the public. OMB Circular A-l 1, Preparing and Submitting Budget Estimates, lists requirements for preparing and submitting IT budget estimates, including requirements to evaluate full life cycle costs, benefits, and Return on Investment. CIO Council Addresses Best Practices and Provides Guidance Federal CIO Council, Capital Planning and IT Investment Committee, Implementing Best Practices, dated June 1998: The 24 major Federal agencies participated in a Best Practices Workshop highlighting their approaches for selecting, controlling, and evaluating critical IT investments. A Practical Guide to Federal Enterprise Architecture, Version 1.0, February 2001: This guide states that an Enterprise Architecture establishes the agency-wide road map to achieve an agency's mission through optimal performance of its core business processes within an efficient IT environment. The Chief Architect, in conjunction with the CIO and select Agency business managers, defines the architectural principles that map to the organization's IT vision and strategic plans. As shown in Figure 1, architectural principles should represent fundamental requirements and practices believed to be good for the organization. Strategic Plans I tip licat ions EA Policies and Guidelines ¦ EA Development - tA Use ¦ EA Maintenance ¦ EA C ompliance IT Visiuxi, Rjtnj 'Ml L'lLlL'lUv hniI Pmri.irm Business INeeds Principles - F.A - F.t i I.i-tt j iti :-:r Systems Life Cycle Systems Migration Technology Insertion Dual OjjtnaLiuiii D1-31I 1 iyrriMTil. PIhtis Actions rnpit^l Planning mill Investment Control - Pi uj eul SeleuLiun ¦ Pt 1 ij ri :l. C, 1 itiI.ti i1 ¦ P rnj rr.t. F.val i L=it.i n n ¦ Return on Investment Figure 1. Role of Architecture Principles 3 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act EPA Delegation for CIO EPA Delegations Manual 1200, 1-84, Information Resources Management, dated December 18, 2001, specifically requires the CIO to: (1) Approve the Agency's IRM Strategic Plan, Five-Year IRM Implementation Plan, IRM investment portfolio, and IRM contracting strategy; (2) Establish policies and procedures for the management and security of records, files, data, and information systems and technology; (3) Approve the acquisition of information technology resources; and (4) Establish and maintain a continuing program for the management and security of records, files, data, and information systems and technology. Authorities (3) and (4) above were re-delegated on June 13, 2002, to OEI's Director for Technology Operations and Planning. These authorities may be re-delegated further to Assistant Administrators, Regional Administrators, the Chief Financial Officer, and other senior Agency officials. Moreover, these officials may further re-delegate authorities within their respective organizations. EPA Requirements for Software Development EPA Directive 2100, IRM Policy Manual, establishes a policy framework for IRM programs at EPA. In particular, Chapter 17, System Life Cycle Management, identifies life cycle requirements for information systems projects. These requirements include the System Management Plan, cost-benefit analysis, and a risk analysis at each stage of the system development life cycle. Chapter 17 also prescribes that a system charter be developed during project initiation, including an estimate of life cycle costs, and identifying the appropriate management levels for approving decision papers. A System Management Plan decision paper should be produced at the conclusion of the analysis stage and should be updated as the project progresses. Scope and Methodology We conducted this audit at EPA Headquarters in Washington, DC, starting in January 2001 and issued a draft report in April 2002. Subsequent to the draft report, we updated portions of the findings to reflect recent Agency accomplishments. We performed our audit in accordance with the Government Auditing Standards, as issued by the Comptroller General of the United States, and included such tests as necessary to complete our objectives. Exhibit 1 details our scope and methodology, as well as prior audit coverage. 4 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Chapter 2 CIO Needs to Fully Implement Clinger-Cohen Act Requirements EPA's CIO needs to demonstrate strong leadership by providing IT technical expertise and a workable investment management structure to ensure the Agency's many program offices implement the IT capital investment process envisioned by the Clinger- Cohen Act. While EPA has taken steps to implement Clinger-Cohen functions, many aspects continue to evolve, with plans, policies, and guidance still in development. EPA did not effectively manage its IT investments from an Agency-wide perspective; however, it recently established a Chief Technology Officer to provide leadership and implement a comprehensive IT investment program. For the period under review, we found that program officials were still operating under invalidated IT acquisition policies and procedures that allowed them to individually make investment decisions. EPA appeared to be using a slowly evolving, volunteer-based, and decentralized approach to developing, supporting, and managing IT capital investments. In addition, the lack of a monitoring process allowed projects to be executed without a minimum level of management controls. Finally, some program managers did not take the Act seriously and viewed the Agency requirements as another step to satisfy the annual OMB budget call. CIO Relies on IT Budget Instead of Investment Portfolio Process The CIO used the Fiscal 2002 annual budget call to plan IT investments. The Act intended that the CIO establish a performance-based system for implementing, monitoring and evaluating IT projects. The Agency's IT investment process was primarily a budget reporting process. It was used to meet OMB IT program annual reporting requirements and to recommend an annual budget for major systems investment projects. Financial management, procurement, and project management controls were not adequately integrated into the Agency's CPIC process. Moreover, project management practices were inconsistent throughout the Agency. Numerous examples demonstrated that the peer review used objective, yet constantly evolving, criteria for evaluating investment risk. While the peer review process adequately quantified and documented risk determinations, we could not substantiate the basis for Information Investment Subcommittee's (IIS) decisions to (1) lower the risk determinations assigned to some investment proposals, and (2) make recommendations for funding them to the Quality Information Council and CIO. Investment Portfolio Structure Missing Fundamental Elements In 1998, EPA established a CIO position. In 1999, EPA created the OEI and reorganized its IRM structure. However, more than 5 years after implementation of the 5 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Act, EPA still had not sufficiently implemented some fundamental elements of a centralized investment portfolio structure (strategic IRM plan, CPIC process, Enterprise Architecture, and cost accounting process). Specifically: • Senior Agency program managers continued to use outdated and unauthorized IT investment practices. Policies and procedures, such as EPA Directive 2100, need to be revised to incorporate new CIO responsibilities relating to IT procurement, systems development life cycle, project management, cost accounting, and budget. • EPA's IRM Strategic Plan dated back to 1994, and did not reflect Clinger-Cohen Act requirements. However, on July 29, 2002, the Agency updated the plan and issued the EPA Strategic Information Plan: A Framework For The Future. • Leadership and organization for developing the Enterprise Architecture changed significantly over the past two fiscal years. / Until the fall of 2001, the Agency budget submission included the architecture project as a component of infrastructure proposals and, as such, was under that leadership. In its fiscal 2003 budget submission, EPA identified it as a separate architecture project and intensified efforts to complete the baseline and target architectures. / In February 2002, the CIO announced a Chief Architect position to manage the development of an Enterprise Architecture. Then, in May 2002, the CIO established a Chief Technology Officer position to coordinate, implement, and advise on numerous IT investment management activities, including the Agency's architecture. Also, through EPA's CPIC policy, the Deputy CIO for Technology (DCIOT) was assigned responsibility for establishing and publishing standards and procedures for the Agency Architecture, E-government activities, and IT planning. These are positive actions, but the Agency has not yet established a formal chain of command from the CIO to the Chief Technology Officer, DCIOT, and Chief Architect. Formalization of the Chief Technology Officer and Chief Architect positions would help ensure sufficient management authority and resources to implement the Act. / EPA believes it will be able to complete the Enterprise Architecture baseline, target, and sequencing approach by October 2002. However, we have not reviewed the recently-completed draft baseline, and have not evaluated whether available resources will enable the Agency to achieve this milestone. • Senior managers could not obtain timely and accurate cost, benefit, and performance information on IT projects. In 2001, EPA purchased a service level agreement to use off-the-shelf software called Information Technology Investment Portfolio System (I-TIPS), a federally-sponsored software product, for monitoring and 6 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act evaluating IT projects in the CPIC process. EPA indicated it has assigned resources for implementation, developed milestones for production, and will use the software to generate automated reports to OMB for the 2004 budget submission. Furthermore, management states that I-TIPS will be expanded agency-wide in 2003. • Actions are needed to strengthen IT project management controls. Program managers used inconsistent management tools, and EPA had no standard project cost accounting system for providing useful data to project managers. Managers used outdated cost-benefit assessments or chose to omit the assessment as part of the system development process. Moreover, the CIO had not established monitoring or evaluation processes to ensure major information systems proceeded in a timely and cost-effective fashion, met user requirements, and delivered intended benefits to the Agency and affected public. These issues are covered in greater detail in Chapters 3 through 6. EPA's Process Creates Unacceptable Risk for IT Investments The absence of a fully-developed, centralized investment portfolio structure resulted in management's: • inconsistent and undocumented evaluations - IIS approval of IT investment proposal projects which were documented as high risk by a peer review process, • inability to effectively monitor IT system development or enhancement projects' schedules and costs, • omission of investment benefit evaluations for completed IT projects, and • inability to document and account for IT project investment costs. The slowly evolving and decentralized approach that was being used to develop an IT investment control structure was not successful. EPA's approach allowed IT projects to be funded without proper justification, and in the absence of adequate management controls. EPA invested resources on outdated systems that did not maximize the efficiency or resolve long-standing problems, such as integration of environmental data. For example, the Air Quality System spent over $8 million from fiscal 1996 through 2001 for the project's Phase 1. The fiscal 2001 budget submission for the project included a statement of intent to make modifications in Phase 2 to adapt the system to function with EPA's Central Data Exchange portal and incorporate Agency data standards. However, these critical functional modifications were not addressed until fiscal 2002, about 6 years into the project. 7 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Conflicts between the EPA Delegation 1-84 and prior procurement policies caused program and regional managers to award new IT contracts without proper CIO approval. Also, existing IT contracts, with a maximum value totaling approximately $1.6 billion, can be awarded new work without proper authorization. Under EPA Delegation 1-84, the CIO is the only manager authorized to approve acquisitions of IT resources. In June 2002, this authority was re-delegated to OEI's Director for Technology Operations and Planning. This authority can be re-delegated further. However, this delegation conflicts with and invalidates prior EPA procurement policies and practices in EPA Directive 2100. Overall, there is a high risk that EPA's technology investments will not result in significant improvements in organizational efficiency and productivity, or enable EPA to work better with states, tribes, local governments, private industry, and the general public. EPA planned to spend approximately $449 million for IT investments in fiscal 2002, so poor investment choices could have significant monetary ramifications. To avoid risk, EPA must ensure that its target enterprise architecture is fully integrated with its Government Performance and Results Act goals and objectives, IRM Strategic Planning, and IT acquisition processes. Until this integration is achieved, EPA will continue to struggle with its ability to reinvent organizational processes, integrate and manage data, and build a scalable and reliable network architecture. In its fiscal 2003 budget submission, EPA took the first step in consolidating duplicate systems when it combined four modernization efforts into two investment proposals. EPA's process for evaluating investment proposals appears to consider data standards requirements and system duplications; however, management must continue to strengthen procedural controls to minimize effects of a weakly integrated process, such as: • IT investments that are not driven by business priorities and mission goals, • investing in stovepipe and duplicate systems, • IT investments that do not take advantage of technology advances and reduced costs, • inefficient reporting processes for states and private industry users, • application systems that do not comply with environmental data and interoperability standards, and • not meeting increased public access and security requirements. Until EPA fully implements the Act's requirements, management will be unable to make fully-informed decisions regarding IT investments. 8 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Strong CIO Leadership Needed to Implement and Enforce Act Although it has been more than 5 years since the Clinger-Cohen Act was implemented, EPA has yet to comply fully with its statutory requirements. We believe this was due, in part, to the fact that EPA did not have a presidentially-appointed and Senate-approved CIO prior to December 2001. Although EPA reorganized its IRM office and established a CIO position, there was little change in the Agency's IT operations or investment practices until recently. The lack of strong CIO leadership and a comprehensive investment portfolio structure perpetuated the Agency's unsuccessful, decentralized IT investment process. The CIO should target key agency-wide problems through the CPIC process (e.g., integration of environmental data, electronic reporting, duplicate systems, Geospatial Information, data standards, and data management). The new CIO's actions show that she agrees. For example, EPA used the CPIC process findings to stop operating funds for the Geographical Information Systems' investment. Recommendations We recommend the Chief Information Officer: 2-1. Assign sufficient resources and expertise to ensure timely and effective implementation of report recommendations. 2-2. Continue with strategy to develop and execute a comprehensive, prioritized, multi-year plan to address gaps and bring EPA's IT policy collection to the "should be" state. In particular, the plan should include appropriate practices for the Enterprise Architecture, CPIC process, and IT acquisitions addressed in the Clinger-Cohen Act, OMB guidance, and EPA Delegation 1-84. 2-3. Continue to work with the Director for Acquisition Management to (a) direct contracting officers and other procurement personnel to only accept procurement requests with a formal CIO approval or officially re-delegated procurement authority; and (b) establish interim delegations, policies and procedures for IT procurement, until formal re-delegations are revised and implemented. Agency Response The CIO agreed overall with the emphasis placed on establishing an effective IT resource investment program. However, the CIO identified specific findings and recommendations that the CIO did not believe reflected recent Agency accomplishments. 9 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act OIG Evaluation We made changes to the report findings and recommendations based on the CIO's response, acknowledging that accomplishments not previously noted were due to (1) recently-completed actions, and (2) EPA's evolving IT investment process, procedures, and selection criteria. While we updated the report's information based on management's comments, we believe significant issues still need to be addressed to institutionalize the Act's requirements. Establishing Agency policies and procedures is only the first step. Monitoring and evaluating IT investments against a set of minimum, critical criteria can ensure the institution is operating as desired for IT capital investments. Furthermore, formalizing the Chief Technology Officer and Chief Architect authorities and responsibilities should help ensure adequate resources are dedicated to the completion and maintenance of the Enterprise Architecture. Then, monitoring and evaluation of IT investments can provide a basis to recommend modifications to the Agency's Enterprise Architecture. The CIO has established an ambitious schedule to address this report's recommendations and, to succeed, EPA will need to continue dedicating significant resources for planning, procuring, monitoring, and evaluating IT investments. 10 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Chapter 3 Weaknesses in CPIC Process Place EPA's IT Investments at Risk The Agency's CPIC process was inadequate to properly manage EPA's IT investments. Most of EPA's major fiscal 2002 IT investment proposal projects are high risk and operating with little oversight. Moreover, projects are not evaluated upon completion. In total, the fiscal 2002 budget submission indicated EPA was planning to spend $449.4 million for IT investments, including $203.2 million for major projects. EPA's fiscal 2002 CPIC investment portfolio process was primarily a peer review risk assessment process that: used constantly evolving Agency-wide priorities for selection, provided little oversight of individual projects' execution during the Control phase, and did not evaluate the adequacy of completed projects in an Evaluation phase. EPA's fiscal 2003 CPIC process was basically the same. As a result, as discussed in Chapter 2, the Agency may have invested resources on outdated systems that did not maximize the efficiency or resolve long-standing problems, such as integration of environmental data. Numerous Documents Provide Federal Guidance OMB provides the primary Federal guidance in Circular A-130, Management of Federal Information Resources; Circular A-l 1, Preparing and Submitting Budget Estimates; and Circular A-94, Guidelines and Discount Rates for Cost- Benefit Analysis of Federal Programs. The CIO Council and General Accounting Office (GAO) have both published additional Federal guidance that describes the process. GAO provides an illustration of this process (see figure 2) in Information Technology Investment Management: An Overview of GAO's Assessment Framework (Exposure Draft), GAO/AIMD-00-155, May 2000. HiwdoyiOLi ton1 you haws selected the be?projects? ;creen Rank Select Select Phase 7 Hw are you snswing MprojKti Mivet benefits'? Evaluate Phase Contact renews Control Phase Monitor process Take corrective actons Make adjustments Apply lessons learned Are the systems dehmifg tWiaf }Ulj expected? Figure 2. IT Capital Planning and Investment Control Process 11 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Existing CPIC Process Inadequate to Manage EPA's IT Investments EPA's IT CPIC process did not adequately select, control, and evaluate the appropriate mix of IT capital investments using objective, risk-based criteria consistent with the Agency's Enterprise Architecture and IRM Strategic Plan. Under the current process, EPA's Chief Financial Officer prepares three exhibits (52, 53, and 300b), at varying times of the fiscal year, for EPA's annual IT Budget submission. OEI's Information Investment Subcommittee (IIS) considers the results of an annual risk assessment review of the major investment proposals listed in Exhibit 300b and, during the Select phase, makes funding recommendations to the Quality Information Council and CIO. However, EPA's CPIC process provides little oversight of individual projects' execution during the Control phase and does not evaluate the adequacy of completed projects in an Evaluation phase, as recommended in Figure 2. The peer review risk assessment was the most substantive and documented process that EPA used to objectively manage annual IT investments. However, at the IIS review level, we found a decision process that lacked adequate evidence to (1) substantiate subjective executive decisions that differed from peer review recommendations, and (2) describe how discrepancies identified by the peer review were resolved. As such, Agency management planning and budgeting recommendations for fiscal 2002 appeared to be based on IIS opinion, rather than the objective peer review risk evaluations. While the peer review process objectively quantified and documented risk determinations, we could not adequately substantiate the basis for the IIS votes which lowered the risk assigned to investment proposals by the peer review process. Nevertheless, the IIS recommended funding the proposals to the Quality Information Council and CIO. The same basic CPIC process was used for EPA's fiscal 2003 IT Budget submission, although specific criteria for the peer review process changed. Our review of the three specific phases disclosed the following: Select Phase Recommendations Not Supportable or Justified. Many IIS recommendations were not supportable based on objective criteria. We evaluated information from EPA's Exhibit 300b IT budget submission, the major IT project document; OMB's risk analyses of that submission; and EPA's internal CPIC Peer Review risk assessment. From those sources, we summarized the investment proposal responses, focusing on 4 key risk factors for the 48 major IT proposals listed in EPA's fiscal 2002 budget submission to OMB. We compared the 48 investment proposals to the results of EPA's CPIC Peer Review risk assessment, OMB's risk assessment report card, and our assessment for the 4 key control areas. OMB clarified that they considered projects to be high risk if they did not demonstrate compliance with key requirements, or the information provided was not sufficient to determine the risks. OMB's risk assessment report card reflected that, overall, 89 percent of EPA's major projects were high risk, 12 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act while EPA's Peer Review assessed that only 8 percent were high risk. Our assessment concluded that all 48 proposals were high risk, based primarily on the fact that the Agency had not provided an Enterprise Architecture for IT managers to use in preparing IT investment proposals. In spite of not having an Enterprise Architecture, all the proposals nonetheless indicated they were aligned with an Architecture. Details on our comparison are in the following table. EPA Major Investment Proposals Key Project Risk Factors (Fiscal 2002) Key Risk Factors OMB Assessment 1 OIG Assessment Peer Risk Assessment Percentage of IT projects not aligned with EnterDrise Architecture 100% 100 % N/A Percentage of IT projects not including adeauate security Dlannina or when not clear 4 % 33 % 56 % Percentage of IT projects not including a comDleted current cost-benefit analysis or when not clear 100% 56 % 40 % Percentage of IT projects not having aDDroved svstem manaaement Dlan or when not clear N/A 48 % 42 % Percentage of High-Risk IT Investment Proposals 89 % 100% 8 % The IIS reviewed the internal risk assessments and agreed with the conclusions that some of these projects were high risk. Nevertheless, the IIS recommended to the Quality Information Council and the CIO that all 48 projects be recommended for funding in the fiscal 2002 budget submission. OEI told us that these projects were recommended for funding only after substantial corrective actions were taken to make the business case, and a fourth review of the project proposal was conducted. 1 We calculated percentages based on raw data (# of projects) and footnote information associated with the "major" projects (Steady State, Mixed and Development/Modernization/ Enhancement), as taken from documents provided to EPA by OMB regarding the Agency's fiscal 2002 IT budget submission (dated July 26, 2001). 13 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Significantly Deficient Projects Recommended for Funding. In spite of the risk assessment process, all the projects with significant weaknesses were recommended for funding in the fiscal 2002 and 2003 IT budget submissions. For example, major projects were found to have significant weaknesses by the peer review process. The IIS downgraded these projects from "red light" to "yellow light" in fiscal 2002, but we found no evidence of how the significant deficiencies were resolved. In fiscal 2003, the peer review process once again stated these projects contained significant weaknesses. Once again, the IIS recommended them for funding. The documentation provided did not contain clear, objective evidence from which we could conclude whether the cited deficiencies had evolved during the 2-year span or simply remained unchanged. Our analysis was confined by the fact that the risk assessments used different documentation and evaluation requirements each year. The CPIC process should rely on one minimum set of consistent objective criteria applied throughout all levels of the selection review hierarchy. Inconsistencies Noted. The narrative for the CPIC IT budget submissions were unclear about the Enterprise Architecture and conflicted with the Agency's fiscal 2002 Annual Performance Plan goals. For example, EPA's key architectural project, the Information Integration Program, refers to the Integrated Compliance Information System (ICIS). The Enterprise Architect document states that ICIS is "being designed to interface with only a few ... legacy systems, but the technology is scalable ..." However, the fiscal 2002 Annual Performance Goals discusses ICIS in terms of 14 existing systems. From these conflicting perspectives, it was not clear how the existing legacy systems were to be integrated with the Enterprise Architecture strategic framework. Control Phase EPA was not monitoring the execution of IT capital investment projects during the year, thereby preventing the CIO from adequately managing ongoing IT investment projects. In fiscal 2002, OMB established baselines to measure progress and performance for projects' scheduled milestones and costs estimates. OMB required that agencies explain schedule slippages and increased costs greater than 10 percent. EPA reports this information in annual Exhibit 300b reports. However, common industry practice is to use a 4 week time frame for monitoring and measuring variances from the project plan. In our opinion, the Agency should monitor the execution of its projects through periodic reports (at least quarterly) that managers can use to identify emerging cost or schedule problems and initiate compensating actions. Evaluate Phase The Agency did not perform any post-implementation reviews or evaluations of completed IT projects. EPA's OEI has taken steps to implement a Post-Implementation Review Phase. In addition, management prepared a list of completed or terminated 14 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act projects that would require review for the first time during the fiscal year 2003 CPIC process. CPIC Management Problems Stem from Several Causes Many factors have contributed to the ineffectiveness of EPA's current CPIC process, as discussed below. CIO Needs to Institutionalize a CPIC Process In June 2002, EPA issued EPA Order # 2100. A. 1 to formally recognize CPIC policies in the Agency Directives. As a next step, the CIO needs to establish Agency-related CPIC procedures and guidance. Insufficient Staff Dedicated to CPIC Process In our opinion, the CIO had not dedicated sufficient resources to administering a fully functional CPIC process. The lack of administrative and financial resources restricted EPA's capability to implement a comprehensive system for managing its IT investment portfolio. The Agency's IT program for fiscal 2002 totaled $449 million. Yet, the CIO only established two full time positions (team leader and one staff) as the primary resources to implement and execute an EPA CPIC process. The permanent positions were supplemented by an ad hoc team for the peer risk assessment and the review of proposals by the IIS. EPA should assign sufficient resources and expertise to address IT acquisition and development. Implementing l-TIPS Would Structure CPIC Process Implementing the Federally-sponsored I-TIPS software, an automated investment control and reporting system, would provide EPA with a valuable tool for monitoring and managing its IT investment portfolio. This tool already is being used by more than half of major Federal agencies. Implementing I-TIPS would help EPA select IT proposals, monitor the execution of funded IT projects, and electronically report IT investment submissions to OMB. Although EPA's OEI appeared to seriously consider using I-TIPS, during the review cycle, management could not provide evidence to support that they planned to implement the software product in the near future. In March 1999, OEI conducted a study, Report on the Results of I-TIPS Process Analysis and Feasibility. Then, in 2001, EPA purchased a Service Level Agreement for I-TIPS. In response to the draft report, OEI indicated that it would use I-TIPS during the current budget cycle for generating reports to OMB. Agency-wide implementation of the product is tentatively scheduled for the fiscal 2005 budget cycle. 15 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Recommendations We recommend the Chief Information Officer: 3 -1. Assign sufficient staff to develop a formal manual for the CPIC process in the EPA Directives system, and cross reference it to updated IT policies in Directive 2100 on budget, management, procurement, and the System Development Life Cycle. At a minimum, the manual should include: (a) a description of how IT investments are linked to the Enterprise Architecture and IRM Strategic Plan, (b) a minimum set of mandatory objective, risk-based criteria for use by both the technical peer review and the IIS review for the Agency's IT investment portfolio. (c) performance measures for monitoring and evaluating progress on IT investments, and (d) provisions for post-implementation review and evaluation of IT investments. 3-2. Direct the IIS to not recommend funding IT projects identified by the Peer Review process as having significant weaknesses (i.e., do not meet the minimum established requirements) or duplicating existing projects, until critical deficiencies are resolved and the resolution steps adequately documented. In addition, IIS should clearly document how all risk weaknesses identified by the peer review are addressed and/or resolved prior to the Subcommittee making recommendations to fund projects to the Quality Information Council and CIO. 3 -3. Direct the Information Investment Subcommittee to monitor the execution of IT projects during the fiscal year (at least quarterly) to identify emerging cost or schedule problems and initiate corrective actions. 3-4. Initiate a formal process with written evaluations of ongoing, completed, and terminated information technology projects to evaluate whether the projects or systems are successfully delivering promised benefits at an acceptable cost. 3-5. Complete implementation of an automated portfolio management system (e.g., I- TIPS) to provide timely, reliable information for investment decisions. Agency Response The CIO's response noted that OEI has issued formal criteria for the CPIC process each year since the requirement began. The CIO also stated that EPA used a highly structured approach for its annual data call, although that process continued to evolve 16 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act from year to year. Lastly, the CIO indicated that EPA expects to integrate updated OMB Circular A-l 1 requirements and the Agency's Enterprise Architecture into the next IT investment review cycle. OIG Evaluation Based on the CIO's response and additional discussions with management, we amended the report and its recommendations. The primary area of confusion relates to our use of the terms 'formally establish' and 'structured process.' We agree that EPA annually issued formal guidance and criteria for the annual budget data call for the years under review. The use of an annual data call may be structured for that one year, but evolving criteria from year to year does not provide an adequate baseline for evaluating progress from year to year. Also, this was the first time the CPIC process used a risk-based process, and it was for the purpose of producing risk-ranked budget data. However, the Act intended a portfolio management process, not simply a risk-ranking of projects in the annual budget data call. We modified the report to clarify our intent for the phrases 'formally establish' and 'structured process.' Generally, our concern was the need for formal policies and procedures to establish a consistent management structure. Without this management structure for capital investments, EPA cannot establish a consistent baseline to evaluate and prioritize IT projects over several years. This minimum baseline information is critical for the CIO, IIS, Quality Information Council, and program managers when comparing IT investments, preparing IT investment proposals, accumulating project costs, monitoring the execution of IT investment projects, and evaluating completed projects. 17 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act 18 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Chapter 4 EPA Needs to Organize and Integrate Planning for IT Investments EPA's ability to organize and integrate planning for IT investments depends on the quality and timing of several important factors. EPA must ensure that the Enterprise Architecture is fully integrated with the Agency's Government Performance and Results Act goals and objectives, IRM Strategic Plan, and IT acquisition processes. Otherwise, EPA will continue to struggle with its ability to reinvent organization processes, integrate and manage data, and build a scalable and reliable network architecture. Although EPA has made some progress in developing an entity-wide Enterprise Architecture, the Agency needs to do more to organize and integrate planning for IT investments. For example, numerous essential components of the Enterprise Architecture have not been fully addressed or integrated. EPA's fiscal 2003 and prior IT investments were not driven by business priorities to result in organizational improvements. However, for the fiscal 2004 budget cycle, EPA's Enterprise Architecture Team has provided guidance and worked closely with proposal preparers. Background During 2001, EPA completed many actions towards establishing a baseline enterprise architecture for IT planning purposes. In April, EPA provided OMB with documentation of EPA's first Enterprise Architecture, dated March 29, 2001. The document was not provided to EPA program offices until an Agency-wide conference in July 2001, about 2 months after the IT investment proposals for the fiscal 2003 budget submission were submitted for the Agency CPIC review process. Furthermore, when the OIG met with EPA's Office of Acquisition Management in October 2001, neither the IT Contracting Officer nor the Procurement Office were aware of the document. OMB reviewed the Agency's fiscal 2001 IT Investment Portfolio and noted that they could not match the projects in the proposed Enterprise Architecture to the portfolio. In August 2001, OEI established a workgroup to identify and verify EPA's business processes for the Enterprise Architecture baseline. The work group's efforts occurred after completion of our field work, although we were informed that the group is updating the business processes and aligning them with OMB's Business Reference Model. 19 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Executive Buy-in and Management Controls Required The Chief Information Officer Council recognizes the importance of executive buy-in and support to the IT investment process. The Council also states that an organization should create an architectural team to define and integrate the components. The enterprise architecture is an expansion of the IRM strategic plan that provides an enterprise view of information technology in the context of EPA's business environment. The enterprise architecture defines the current and target (future) components. A transition plan sequences the evolution from current to target. As such, the enterprise architecture should be a document that is continuously modified and maintained to reflect the Agency's current baseline and target business practices, organizational goals, visions, technology, and infrastructure. Figure 3 below depicts the major components of the Enterprise Architecture that must be addressed to accomplish EPA's strategic goals and perform its business. M \ / Techrclogy Figure 3. Enterprise Architecture Framework Various Components Essential to Quality of IT Planning EPA's ability to organize and integrate planning for IT investments depends on the quality and timing of several important factors. Clearly defining the Enterprise Architecture is particularly important because it provides the conceptual framework for integrating the Agency's information technology environment and core business processes to accomplish strategic goals. In the following subsections, we present issues that EPA management must address to ensure the integrity and effectiveness of its IT investment planning system. 20 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act IRM Strategic Plan Goals Need to be Incorporated into the Enterprise Architecture EPA needs to incorporate the updated IRM Strategic Plan goals into a target enterprise architecture. During our review, EPA was severely criticized by Congress, National Academy for Public Administration, GAO, and environmental and industry groups for not having such a plan. On July 29, 2002, the Agency completed its revised plan: EPA Strategic Information Plan: A Framework For The Future. EPA Has Yet to Fully Baseline its Business Processes As of the end of field work, EPA had yet to fully baseline and validate the Agency's business processes essential for establishing a portfolio for future IT investments. EPA's draft Enterprise Architecture document included very high-level business processes; however, these processes had yet to be validated by the responsible program offices. We were informed that some of these business processes have been revised, but were unable to substantiate whether the applicable program offices formally endorsed the work group's conclusions. EPA understands the importance of this activity, and plans to perform a validation process this year. Draft Enterprise Architecture Baseline Security Architecture Needs to be Expanded Although OEI's draft baseline Security Architecture addresses many pertinent risks in EPA's Security program, it does not adequately address two important components: facility physical security and personnel security requirements. The Enterprise Architecture document states the Agency maintains a security infrastructure of approximately 1,600 servers for network support, application hosting, scientific computing, and graphics. OEI centrally supports these servers. The document also indicates that the Agency owns an additional 900 servers not supported by OEI personnel, but it does not adequately address who supports these servers. OEI confirmed that these servers store sensitive data. Therefore, the physical and personnel security requirements of these servers need to be added into the baseline security architecture. Key Data Needs to be Developed, Analyzed, and Controlled As shown in Figure 3, the Enterprise Architecture conceptual framework should consist of five components. As such, the Enterprise Architecture should define mission-critical data needs to properly support the IT investment process. However, the draft Enterprise Architecture plan we reviewed did not (1) specifically recognize (i.e., require) 21 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act individual Agency data standards and related metadata2 baseline information, and (2) adequately address other critical data used by stakeholders and programs business processes. EPA states it will address program-specific data needs across several dimensions. As of the end of field work, EPA had approved six Agency data standards, and recently it adopted a seventh standard. In addition, the Office of Water had implemented some program data standards. Although these efforts were underway, EPA's intended infrastructure for managing and sharing environmental data did not adequately address how EPA's program users and stakeholders were to use existing and future data registries to manage data. In fact, this issue has been a long-standing OIG concern, as noted in a prior report, Information Resources Management: Office of Water Data Integration Efforts (No. 8100177), dated June 22, 1998. We had recommended that EPA support its data standards program by using the Environmental Data Registry as a central repository for publishing and recording Agency data standards. The Enterprise Architecture Plan we reviewed did not incorporate this recommendation. However, EPA states that its current draft version of the Enterprise Architecture clearly describes the registry as a critical component of its target architecture. In its draft Enterprise Architecture, EPA recognizes that more detailed descriptions of critical data are necessary. Among other things, EPA will need to validate the information flow and relationships, as well as data descriptions and relationships, described in the initial Enterprise Architecture. Without this step, EPA cannot begin to establish a target architecture and define the required sequencing plan for migrating from the baseline to the target architecture. Complete Inventory of Systems Needed for Enterprise Architecture EPA needs to complete an update of its inventory of general and application information systems. This baseline of systems should identify current critical business processes, related systems (major and significant), and mission-critical data in those systems. At that point, the baseline can be used to identify IT investment projects that will meet the Agency's current needs, eliminate redundant systems, and build an IT structure to accomplish EPA's goals. However, we noted a number of inconsistent inventories. EPA's March 2001 submission to OMB included a Year 2000 Systems Inventory that listed 70 major and significant application systems. However, the Enterprise Architecture, dated March 2001, only listed 46 major systems. In September 2001, the CIO reported to OMB in its On Implementation of the Government Information Security Reform Act report that it had 189 systems. In its response to the draft report, OEI stated the Enterprise Architecture will incorporate all systems into an Information Resources Registry System, which is scheduled to be operational by the 2 Explanation of specific data fields, including information regarding its source, collection method(s), and in what context the data can be used. 22 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act end of fiscal 2002. OEI did not indicate how long it would take to fully populate the Registry System. OEI also plans to link the Registry System and the Enterprise Architecture. In addition, the Enterprise Architecture document states that sufficient information on Agency application interfaces is not available. The document states the CIO plans to gather and document this information as part of the Agency's ongoing application inventory initiative, including documentation regarding major interfaces with applications outside of the Agency. For example, this year, EPA intends to gather more information on internal system interfaces and partner interfaces within the framework of its National Environmental Information Exchange Network. Enterprise Architecture Needs to Address Scalability of Virtual Private Network The draft Enterprise Architecture does not adequately address EPA's existing and future technology components for its next-generation wide area network. The Agency needs to address "scalability" and Virtual Private Network (VPN) concepts to grow with the Agency's evolving needs. Scalability refers to the ability to expand a network to accommodate future needs; a VPN is an electronic network, without physical limitations, specifically designed to secure transmissions. With regard to scalability, the Enterprise Architecture document did not explicitly identify minimum response times for key transaction-based systems and for business application systems on the Agency's wide area network. Moreover, EPA's July 2001 Network Requirements Study indicated that bandwidth utilization for some circuits experienced bottlenecks for certain portions of the network and responsiveness for newer systems ranged from very poor to good. Also, whereas management has recognized the need for virtual private networks, they only reference it in light of long-term needs. We believe the VPN concept is needed today to help the Agency comply with existing Federal telecommuting statutory requirements and to satisfy current business needs. We agree with Agency officials that technical issues, such as transaction response requirements and scalability, normally are addressed in a Technical Architecture. OEI's response to the draft report mentioned a "Technical Reference Model" and, we agree, that may be a suitable planning document in which to address these issues. OEI agrees with the importance of secure external communications and states they will take critical steps to start implementing VPNs next year and, pending available resources, will make full operations available on an enterprise basis in 2004. Enterprise Architecture Should Address Middleware EPA's Enterprise Architecture should identify the middleware architecture needed to address those client-server systems already implemented, as well as those envisioned and planned to strengthen the overall usability of the distributed architecture. 23 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Middleware architecture includes such things as message brokers, extensible Markup Language, and directory structures used to facilitate interconnection of systems and applications. EPA's draft Enterprise Architecture overlooked this aspect of IT planning, but management may want to address these topics as part of the "Technical Reference Model" mentioned in OEI's response to the draft report. To minimize the risk of incompatible communications, a standard middleware architecture could greatly benefit application developers with a single consistent interface for both inter- and intra- application communications. Various Causes Contributed to Lack of Planning No Central Planning Organization or Appointed Authority EPA's IT planning activities suffered from a lack of a central organization and authority. EPA's IT planning is currently managed using a decentralized and fragmented structure involving numerous individuals and offices. Agency-level coordination was generally accomplished through project briefings to the Quality Information Council and its four subcommittees. With regard to the fiscal 2002 budget process, informal meeting minutes would support that the Council deferred formal management planning decisions in lieu of receiving briefings by numerous project managers and the Council's subcommittees. Also, EPA needs to define the role and authority of its Chief Architect for IRM. The role of this Chief Architect is to oversee development and coordination of the Enterprise Architecture with other planning elements that should materially shape and drive the IT planning structure. The CIO named an individual to this role in February 2002 (via electronic mail), but there has been no formal definition of the position's scope and responsibilities in policy, nor any official delegation of authority. Further, we identified several IT planning-related, Agency-wide documents, projects, and work groups that should be coordinated to ensure their individual visions and plans are aligned. Together they will enable EPA to optimally execute its program goals and deliver environmental and human health improvements. To EPA's credit, management established a central Enterprise Architecture workgroup in August 2001. While EPA has planned activities to coordinate and develop the Enterprise Architecture, management must also establish a permanent central organization with dedicated resources and assigned responsibility to maintain this living document. Agency-wide Enterprise Architecture components need to be addressed and maintained for the following functional areas: the identification of EPA's major and significant systems; defining the security architecture; validating the business processes with program offices; developing the Middleware architecture and defining baseline telecommunication requirements; defining Working Capital Fund capital investments; and approving individual IT project management plans for major projects or systems. 24 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Finalizing Information Integrated Program Plan Needed In its fiscal 2003 budget submission, EPA identified the Information Integration Program as its only major architectural project for deriving and completing an enterprise architecture. As critical as the project is to EPA's Enterprise Architecture development efforts, no final management work plan has been implemented for this project since the draft was issued in December 2000. Management is required to issue a final, approved work plan in accordance with Agency Directive 2100, and should do so to ensure the timely success of the individual program, as well as the overall quality of the Enterprise Architecture Plan and the Agency's future technology investments. The Chief Architect provided information that indicates EPA's program and regional offices will be asked to co-develop the Agency's baseline and target elements for the Enterprise Architecture. With OEI's leadership and facilitation, the program and regional offices will conduct their own architectural needs analysis, and realign their respective systems with EPA's evolving target. During our fieldwork, we were unable to substantiate how this will be accomplished. In OEI's response to the draft report, management assured us that participants have been informed of their roles and responsibilities. In addition, they stated the Chief Architect is developing explicit guidance to formalize roles and responsibilities for regional and program offices. Management also stated that the Enterprise Architecture was scheduled for completion by October 2002. Recommendations As the number one priority, we recommend that the Chief Information Officer direct the Chief Technology Officer to: 4-1. Formally institutionalize: (a) in policy the Enterprise Architecture program to plan, manage, monitor, and control the development and maintenance of the Enterprise Architecture plan. (b) the Chief Architect position by clearly defining and documenting the roles, responsibilities, and authority of the job in policy or through a delegation. Next, we recommend the CIO target the following key actions to complete the Agency's baseline and future plans for the Enterprise Architecture: 4-2. Establish a permanent organization under the leadership of the Deputy Chief Information Officer for Technology to update and maintain the Enterprise Architecture in accordance with the Agency IRM Strategic Plan and its Government Performance and Results Act requirements. 25 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act 4-3. Identify current major and significant general and application systems to establish an accurate inventory of such systems and integrate this information with both the Agency's Enterprise Architecture application component and the IT CPIC Portfolio. 4-4. Complete the project to publish an updated Enterprise Architecture and document the project as required by Agency policy. 4-5. Finish implementing a robust Agency information repository and (o) require the use of the data registry for Agency maintained data, (p) map EPA's data and information resources, and (q) complete on-going efforts to adopt life-cycle data management principles for the Enterprise Architecture data and systems components. The CIO should implement the following recommendations as the Enterprise Architecture is developed and updated: 4-6. Use a top management verification, validation, and approval process to ensure program business processes and goals are accurately reflected and incorporated into the Enterprise Architecture. Subsequently, formalize the process as a discipline for updating the Enterprise Architecture document. 4-7. In coordination with the Office of Acquisition Management, jointly develop an approval process that ensures the Enterprise Architecture concept is incorporated in future IT contract activities for large and significant IT projects. 4-8. As part of a Technical Reference Model or Technology Architecture, address technology components, such as interfaces, transaction response times, and baseline telecommunications requirements to support a scalable, reliable, and secure network infrastructure for the Enterprise Architecture. Agency Response The CIO generally agreed with our recommendations, but believed many actions currently underway were not recognized in the report's findings. OEI had made progress in addressing our concerns and, therefore, the CIO suggested that we revise specific findings or recommendations to reflect recent accomplishments. OIG Evaluation We made changes to the report findings and recommendations based on the CIO's response, acknowledging recently-completed actions and planned activities. We agree that EPA has taken significant first steps to address our report's findings and recommendations on IT planning. However, many actions were initiated after we finished audit field work, and some actions are still in progress. 26 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act We attempted to be as specific as possible in our recommendations to provide appropriate direction and recognize current ongoing efforts. For example, we agreed that some of the technical components can be addressed appropriately in a Technical Reference Model or Technology Architecture, rather than the Enterprise Architecture, and amended the recommendation accordingly. The CIO has established an ambitious schedule to address this report's recommendations, and it will require a significant amount of dedicated resources to not only complete them, but to maintain the EPA's planning structure for IT capital investments. 27 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act 28 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Chapter 5 EPA Needs To Strengthen IT Project Management Controls For the six EPA IT major projects reviewed, we found significant project management control weaknesses, a lack of compliance with Agency system development policies, and inaccurate project status information reported on the Clinger-Cohen budget submission. EPA incorrectly reported an approved System Management Plan (SMP) was being followed for projects. Further, SMPs were either out of date or had never been formally approved and signed. We also found significant variability in EPA's working capital fund expenditures, which adversely impacted the system development project's planning and budgeting activities. Several key factors contributed to the lack of management controls over IT projects: • OEI had not updated IRM policies or established interim guidance to convey new requirements, and project managers did not practice existing policies; • managers were not using a phased, sequential system development process; • EPA had not adopted standard tools for reliably managing IT project information resources, schedules, products, and costs; and • until fiscal 2002, EPA had not provided a means for project managers to track project and contractor support costs. The CIO needs to establish controls to monitor project managers and ensure they use key management controls (e.g., SMPs), and maintain current cost-benefit analyses and project cost records. Otherwise, the CIO has little assurance that IT investment projects represent cost-effective solutions. Primary System Guidance OMB Circular A-130, Management of Federal Information Resources Management, establishes requirements for: preparing and updating a cost-benefit analysis for each information system throughout its life cycle; conducting post implementation reviews of information systems development projects to validate benefits; and establishing an oversight mechanism to ensure major systems development projects proceed in a timely fashion toward agreed-upon milestones and deliver intended benefits. OMB Circular A-l 1, Preparing and Submitting Budget Estimates, required two reports for fiscal 2002 budget submissions: 29 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Section 53. This report summarizes an agency's IT portfolio by listing major and significant capital investments for IT system, infrastructure, and architecture projects. Section 300. This is a separate planning and justification report for each major capital investment with a useful life of 2 or more years. Agencies are expected to establish and measure baseline costs, establish a measurable project schedule, and ensure projects support performance goals. OMB Circular A-127, Financial Management Systems, Parts 6 and 7, address financial system requirements. EPA Directive 2100, Chapter 17, identifies an eight- stage life cycle methodology, and establishes specific thresholds for formal review and approval of an SMP for system development or enhancement projects. Documents Incorrectly Reported In its fiscal 2002 and 2003 CPIC project submissions, EPA managers misrepresented the status of key management documents. We reviewed documentation for three of six selected projects. We could not audit two infrastructure projects because, despite repeated requests, EPA managers did not furnish adequate supporting documentation. The sixth, which was EPA's current architecture project, Integrated Information Project, did not have a current, approved SMP. Following are examples of what we found: SMPs • The SMP for AIRS-AQS (Aerometric Information Retrieval System - Air Quality System) had not been updated since originally prepared in 1996. Maintaining a current and formally approved SMP is important because it discloses significant changes to the system development project and ensures accountability. • As of December 17, 2001, the SMP document for the RCRAInfo (currently defined as the Resource Conservation and Recovery Act Information Management System and Waste Information Needs/Informed) did not include the Assistant Administrator's signature approving the project and key decisions, as required by EPA Directive 2100. Project management attributed the lack of signed hard copies to a reliance on electronic documents and e-mail to manage meeting minutes and decision notes. Cost-Benefit Analyses • Project management stated that, given the modular nature of the RCRAInfo project, cost benefit analyses were performed for each major component rather than for the project as a whole. EPA's fiscal 2003 investment submission for this project disclosed total life cycle costs of $70.5 million, an increase of $40.4 million over previously projected costs. Management attributed the increase to: estimated 30 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act regional and state costs, changes to working capital fund rates, and adding years to the system life cycle. An updated cost-benefits analysis would help determine the most cost-effective strategy for implementing the RCRAInfo investment. The cost-benefits analysis for SDWIS/STATE (Safe Drinking Water Information System/State Version Modernization Effort) had not been updated since 1992, despite many changes in design, functionality, and plans to migrate to a web- enabled application. The outdated analysis erroneously leads EPA management to believe that the original return on investment will still be achieved. An updated cost- benefit analysis should be completed as extra functionality is added to the system, such as the planned integration of SDWIS/STATE into the Agency's Central Data Exchange initiative. Primary Architecture Project Lacks Plan Although EPA's Information Integration Program is the heart of EPA's Enterprise Architecture and planning investment strategy, EPA did not recognize the Program as a separate architectural project until the fiscal 2003 budget submission, provided September 2001. As such, no project plan had been finalized to define the vision, scope, or implementation and cost schedules for this architectural project. The project plan would help management ensure that the intended benefits of this complex endeavor do not outweigh the projected costs, as well as provide specified time frames for completing detailed tasks and products. Project Managers Not Adequately Monitoring Status EPA project managers were not adequately monitoring the execution of IT capital investment projects. EPA's 300b IT investments reports showed that projects consistently did not meet cost estimates, scheduled milestones, and planned performance. We compared planned expenditures for 46 IT investment projects in fiscal 2001 against their corresponding actual costs, and found that 37 percent showed more than a 10 percent increase. Furthermore, the investment reports indicated that 78 percent of these projects experienced milestone slippages greater than 10 percent. The data strongly indicates project managers need better standard management tools. Many Factors Negatively Impact Management of IT Investments Numerous factors contributed to the inconsistency of management controls for IT investment projects. These concerns were voiced by many of the project managers interviewed. IT Project Managers Need Standard Tools For the period reviewed, EPA had not adopted standard project management tools to help managers plan, control, and evaluate IT investment projects and track project 31 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act costs, schedules, and resources. SDWIS/STATE is an example of a project that could have been managed better with the help of a project management tool. Standard project management tools help promote a consistent and uniform approach to tracking and managing all forms of project and contractor support costs. A standard tool helps to reduce the communication gap between contractor support activities and what the Agency reported for this IT investment. IT Projects Not Using A Phased Sequential Project Life Cycle The status of a project is often unclear because project managers do not use a sequential, phased development process to clearly distinguish where one series of system development life cycle activities ends and another series begins. EPA Directive 2100, Chapter 17, requires that system development projects follow a sequential, phased systems development life cycle called the "waterfall" method. This method consists of eight sequential stages. Any planned new functionality should be considered a new project, and a new project also should be established when estimated costs exceed stipulated dollar thresholds. Industry recognizes at least three other models for systems development that are sequentially-phased from a project perspective. These approaches are generally referred to as: (a) spiral, (b) prototype, and (c) rapid application development models. Spiral modeling works as a repeating waterfall approach, with a risk analysis at every stage to determine whether cost overruns, schedule delays, or changing requirements will impact the benefits of proceeding. Prototyping uses existing software and lets a group of users define the system requirements for an organization. Rapid application development is based on reusing and modifying software components until they perform as desired. The projects reviewed did not demonstrate any of these acceptable "phased" software development approaches. Rather, we found that EPA generally used an evolutionary approach in which management continuously added requirements to the overall system development project. For example, the RCRAInfo project was simultaneously in more than one stage of the system development life cycle, and management could not distinguish the cumulative costs associated with one set of activities versus another. The project is very broad and encompasses five program area requirements. In 1999, contractors completed the first system development life cycle stage (i.e., the Requirements Analysis) for three of the five areas, while the two most critical functional requirements remained in the first stage. Despite several years of effort, management was still defining RCRAInfo requirements. Business needs can change based on technology advances, so best practices suggest that requirements be defined in less than 2 years. We believe management should have split the program area requirements into two or more distinct projects, so development efforts could progress in a timely fashion from one stage to the next, and managers could easily track associated costs and schedules. 32 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Evolving Nature of EPA's Exchange Network The evolving nature of EPA's architecture project deterred management from finalizing its formal project plan to ensure the cost-effective and timely execution of the Exchange Network. What is now referred to as the Information Integration Project represents the third iteration of the project, and the objectives and intended outcomes have undergone several revisions. Also, the number of infrastructure projects (e.g., registries) affecting the Information Integration Project have been evolving, and management must clarify the role these supporting projects play. Minimal Assurance that IT Investments are Cost-Effective and Controlled The absence of key decision documents and senior management approval (e.g., cost benefit documents, management decision papers, system management plans) increase the risk that funded IT projects will evolve in an unstructured, untimely, and costly manner. Furthermore, expanding and/or changing original project objectives to incorporate evolving business functions results in confusion, complications for proper cost accumulation, and slipped project development time lines and even system development projects that never come to closure. In addition, if projects are too broad in scope to progress through the life cycle in a timely manner, then what originally was thought to be a cost-effective solution may become a bad return on investment. Further, the lack of project management tools inhibits project managers' ability to provide reliable data on a project's status, and contributes to unjustified delays and unsupported cost overruns on IT projects. Chapter 2 contains additional effects relating to EPA's inadequate oversight processes. Recommendations We recommend the Chief Information Officer: 5 -1. Monitor IT investments to ensure that SMPs are prepared in accordance with Agency requirements, and that they appropriately link the Enterprise Architecture and other planning documents to the Clinger-Cohen Act submission documents. 5-2. As part of a monitoring process, re-evaluate funding for IT investments at least quarterly, to determine if they have exceeded budgeted costs or project milestone schedules by more than 10 percent, and ensure that written justifications sufficiently support continuing the project. 33 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act 5-3. Prescribe that standard tools, such as I-TIPS and project cost accounting, be used for managing projects for software development changes to IT systems and project management. The selected tools should be approved by the Chief Financial Officer as being compatible with the Agency's cost accounting and financial systems. We recommend the Air Quality System Project Manager: 5-4. Update the SMP for the Air Quality System project and obtain the signature of approval of the Assistant Administrator for Air and Radiation at the conclusion of the analysis stage for major and significant enhancements adding new functionality. We recommend the RCRA Information Project Manager: 5-5. Update the Project Management Plan for the RCRAInfo project to make it equivalent to an SMP, for planned system design changes and enhancements adding functionality. In addition, the SMP should be formally approved by the Assistant Administrator for Solid Waste and Emergency Response to authorize the IT investment and to ensure a system of accountability. We recommend the SDWIS/STATE Project Manager: 5-6. Establish an SMP for the SDWIS/STATE project and obtain the signature of approval from the Assistant Administrator for Water at the conclusion of the analysis stage and for major and significant enhancements adding functionality. We recommend the Project Managers for the Air Quality System, RCRAInfo, and SDWIS/STATE: 5-7. Manage project development efforts in accordance with the SMP, as updated, throughout the life cycle of the system, and retain the SMP for reference and review by the CIO or the CIO's designated review official. Agency Response We received comments from several Agency officials in response to this chapter's findings and recommendations. The CIO agreed to monitor IT investments and expected to also establish a preselect phase. However, the CIO stated we had not recognized that the current review process required monitoring a project as part of an annual review. Further, the CIO did not agree that one set of project management tools would be cost effective or meet all projects needs. The Assistant Administrator for Solid Waste and Emergency Response and the Director of the Office of Air and Radiation's Information Transfer and Program Integration Division both disagreed with our conclusion that project management controls were inadequate. 34 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act OIG Evaluation We made changes to this chapter based on the Agency's responses, as well as further discussion with management officials. We had used a judgmental sample of the six different kinds of major IT investment projects, and the sample accounted for over half of the fiscal 2002 major IT projects' budgeted funding. We had completed a limited survey, requested supporting documents, and interviewed key project managers. However, we were unable to complete the survey and had to limit our scope of review because three major system projects did not provide requested information. For the three major system projects completing the survey, we did not (1) review all the individual project's management controls, or (2) determine whether the individual project accomplished the objectives identified in the budget submission. Our review concentrated on project management controls and documentation requirements in OMB Circulars and existing EPA System Development Life Cycle policy. We were able to document inaccurate and/or unsupported information being incorrectly reported by the three major IT system projects in the fiscal 2002 budget. For example, the projects (1) did not adequately address OMB requirements by consistently accumulating costs from year to year; (2) could not support total costs from inception of the project; and (3) could not provide current cost-benefit studies addressing costs, needs, and expected benefits. We also found that the projects could not document compliance with existing Agency and Federal system requirements, such as the development and top management approval of a current cost-benefit analysis. Each project was using a different set of project management procedures for the day-to- day execution of the project. We did not evaluate these local project controls. Still, we believe that if EPA was monitoring the projects' execution (at least quarterly) and evaluating completed IT projects, individual project managers would address these critical management controls. Furthermore, if program managers are compelled to report accurate data for critical management controls (e.g., emerging cost and schedule overruns), then the CPIC peer review process can more accurately assess the risk of successful completion for susceptible IT projects. 35 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act 36 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Chapter 6 Project Cost Accounting System Vital for Planning and Managing IT Investments Although EPA implemented an IT project cost accounting methodology in fiscal 2002, EPA managers previously relied on an inconsistent variety of informal cost accumulation processes and records to oversee and measure progress on individual IT system development or enhancement projects. Even now, the accuracy of captured IT costs depends largely on the ability of non-technical staff to consistently and accurately distinguish how IT costs fit into system life-cycle categories, and to appropriately code funding documents. Accuracy also depends on contractors adequately identifying specific software development costs. Cost Accounting a Federal Requirement Cost accounting data is required by Federal laws, standards, and Agency policies. The Clinger-Cohen Act notes that before an IT investment is made, it is to be evaluated using a risk-adjusted return on investment as well as other specific quantitative and qualitative criteria. OMB Circular A-l 1 defines the life cycle phases to be used for reporting IT costs and budgets. EPA Directive 2100 requires system managers to prepare a needs assessment and SMP before a new system development or enhancement project can be approved. Statement of Federal Financial Accounting Standard No. 10 requires agencies to capitalize the full costs of internal use software. Managers Did Not Have Necessary Project Information Prior to the start of fiscal 2002, EPA did not have a standardized project cost accounting methodology for managers to use in overseeing IT projects and systems covered under the IT CPIC process. In the projects reviewed, we found that managers relied on an inconsistent variety of informal cost accumulation processes and records to identify expenses, assess changes to baseline costs and schedules, and measure progress of individual IT development or enhancement projects. In addition, managers needed a standard project management system to allow them to establish reasonable baselines for projects, including tracking and managing project contractors' costs; accumulating labor, working capital fund, and project hardware purchase costs; and controlling changes to system milestones and documentation. Effectiveness of Interim Accounting Practices Untested EPA's Office of the Comptroller issued interim policies and procedures on accounting for IT activities through Policy Announcement No. 01-10, New Information Technology Accounting Requirements. Effective October 1, 2001, this announcement established a 37 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act standard agency-wide method of tracking IT costs using the site/project field in the existing accounting code structure. The announcement also defined three life cycle categories, as well as IT activities, goods and services, and established processes for capitalizing the full cost of internal use software. The majority of EPA's IT project costs are based on contractor and grant costs. Whether the captured IT costs are accurate will depend largely on the ability of IT Project Officers, Delivery Order Project Officers, and Contracting Officer's Technical Representatives to accurately assemble supporting cost documents, accumulate appropriate project life cycle costs, and input the project costs into the Agency's accounting system by life cycle phases. Especially in the early implementation stages, individuals may not have enough knowledge of the IT projects they manage to consistently and accurately distinguish between the significant and major cost categories (i.e., the preliminary design, development, and maintenance phases). Our concern is compounded by the fact that the three system life cycle categories set forth in the Policy Announcement are inconsistent with the phases described in EPA Directive 2100. OEI and the Office of Chief Financial Officer are participating in an agency-wide workgroup to revise and identify acceptable systems development approaches, resolve current differences in life cycle phases, and develop common definitions across various management programs (e.g., accounting, systems development, Enterprise Architecture, and CPIC process). Until the new practice is audited, we cannot be certain that actual Agency practices will conform with the Policy Announcement, or that successful implementation of the policy will result in effective tracking of IT costs for capitalizing the full costs of internal use software. Ability to Assess and Manage IT Projects Impaired The absence of a project cost accounting system impaired IT managers' ability to efficiently and reliably estimate, manage, and report IT project costs. For example, system managers could not perform reliable cost-benefit analyses of technical alternatives, which is useful for developing a sound system/project management plan. Likewise, IT managers could not maximize the value of or perform risk-adjusted Return on Investment analyses. Furthermore, neither the CIO nor Chief Financial Officer could reliably verify or validate the accuracy or completeness of IT expenses reported by program offices and regions. Therefore, IT investment amounts previously reported via OMB Exhibits 53 and 300b were at significant risk of being incomplete, inaccurate, or inconsistent with prior year disclosures. EPA Asserts System Complies with Standards Despite previous OIG report recommendations to implement a managerial cost accounting system, the Office of the Chief Financial Officer had maintained that EPA's financial management system met Federal accounting standards. While Statement of 38 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Federal Financial Accounting Standard No. 10 prompted the Agency to create a methodology to capture IT costs for "internal use" software capitalization purposes, EPA's current interim cost accounting and related management systems still cannot provide managers with enough basic cost information to accomplish objectives associated with planning, decision making, control, and reporting for their respective IRM program activities. However, on September 24, 2002, the Office of Chief Financial Officer submitted an action plan for Expanding Cost Information at EPA. We will continue to monitor the Agency's achievements as they work with program offices to promote the use of cost information in managing for results. Recommendations Implementing appropriate definitions and controls will require the combined efforts of several EPA program offices. We recommend the Chief Information Officer, Chief Financial Officer, and Director for Acquisition Management work together to: 6-1. Institutionalize consistent definitions of systems life cycle stages and IT costs in Agency policy to be used for contracting, accounting, IT systems, project management, and the capital planning investment control process. We recommend the CIO and Chief Financial Officer work together to: 6-2. Institutionalize in Agency policy consistent systems life cycle and IT costs definitions for revising EPA Directive 2100, and the interim IT activities policy guidance. We recommend the Chief Financial Officer lead an effort to: 6-3. Complete a needs and feasibility assessment of alternatives to determine what types of project cost information and supporting documentation are needed for the capital planning investment control process and managing IT projects. Agency Response Responding for EPA's Chief Financial Officer, the Comptroller agreed in general with our recommendations and pointed out that Policy Announcement 01-10, effective October 1, 2001, implemented IT project cost accounting, which is a new way of conducting business for EPA. Both the Comptroller and the CIO did not agree with a proposed recommendation to amend all current system development contracts to identify system development costs by Agency system development life cycle phase. The Comptroller stated that the policy already requires Project Officers, Delivery Order Project Officers, and Contracting Officer's Technical Representatives to code project costs for projects and systems under their control. 39 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act OIG Evaluation Despite Agency assurances, we still have concerns about whether accurate cost information will be available to permit Project Officers, Delivery Order Project Officers, and Contracting Officer's Technical Representatives to accurately code costs for projects and systems. As the Comptroller pointed out, this is a new process that only was established at the end of our field work. As a result, no information was available to complete a detailed evaluation of operational cost accumulation controls. We have dropped our prior recommendation to amend requirements for existing software development contracts until the fiscal 2002 financial statement audit evaluates the adequacy of this new cost accounting process for accumulating software development costs by project. 40 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Appendix 1 Details on Scope and Methodology We performed our audit in accordance with Government Auditing Standards, as issued by the Comptroller General of the United States. The audit included tests of the program records and other necessary auditing procedures. We began preliminary research on January 16, 2001, and an in-depth review on August 21, 2001. We issued a draft report in April 26, 2002. We conducted this audit at EPA Headquarters in Washington, DC. At the time of our audit, our scope was limited because the Agency could not provide a final work plan for the Information Integration Program project, also known as the National Environmental Information Exchange Network project. Also, we could not substantiate how the Working Capital Fund process integrates with the IT investment process (see Scope Limitations section below). To accomplish the audit objectives, we attended hearings on July 11, 2001, on Senate Bill 803, and documented Testimony before the Senate Governmental Affairs Committee. This bill was to address the need for a Federal CIO to manage IT investments under the Clinger-Cohen Act. We compiled a list of public laws related to IT acquisition and management that affected implementation of the Clinger-Cohen Act. This included the Electronic Government Act, the Paperwork Reduction Act, and the Federal Acquisition Regulation. We reviewed Congressional Reports and noted the problems Federal agencies were experiencing implementing the Clinger-Cohen Act. We reviewed OMB Circulars pertaining to implementation of the Act, and feedback provided by OMB to EPA concerning Agency IT budget submissions. We reviewed the Agency's Enterprise Architecture dated March 29, 2001, and summarized the Federal requirements for developing Enterprise Architecture documents. We researched and reviewed documents issued by the Federal CIO Council relating to the implementation of the Clinger-Cohen Act. EPA has actively participated in the Council's survey and study projects. We reviewed EPA IRM policies related to implementation of the Clinger-Cohen Act We met with Agency personnel knowledgeable of and responsible for writing IRM policies. At the time of our review, EPA had established an Agency work group to address the needed revision of System Development Life Cycle polices to support the requirements of the Clinger-Cohen Act. We reviewed Agency delegations dealing with implementation of the Clinger-Cohen Act to ascertain whether appropriate authority had been delegated to the CIO by the 41 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Administrator, and whether the CIO had delegated appropriate authority to program officials. We consulted with the OIG Counsel on this matter. To gather information on the implementation of the Clinger-Cohen Act in other Federal agencies and determine potential benefits that could be implemented by EPA, we interviewed personnel at three other agencies: Treasury, Housing and Urban Development, and Agriculture. For example, I-TIPS was a tool used by management at these agencies. We interviewed personnel responsible for implementing and managing EPA's CPIC process, including the OEI Director; and personnel in the Office of Technology Operations and Planning and its Information Technology Policy and Planning Division. Division personnel interviewed included the Chief of the IT Strategic Planning Branch and CPIC Team Leader. We also attended various OEI meetings related to the CPIC process. We reviewed EPA's IT budget submissions for fiscal years 2002 and 2003, including various budget proposals. Our review included a comparison of the proposals for the 2 years to determine any proposed changes, the differences in budgeted and actual costs, and the cost variances. We also noted whether the proposal indicated a Cost Benefit Analysis and a Security Plan had been completed. We examined various documents provided by OEI, including budget call letters, instructions for preparers, the organization of the peer review, instructional material for reviewers, proposal evaluation criteria, peer review scoring, ranking and comments, notes, agendas, and actions of the Investment Subcommittee. We reviewed the agenda, notes, and actions of the Quality Information Council. For three IT investment projects, we reviewed the adequacy of information and documentation in support of their Clinger-Cohen Act submission documents for fiscal 2002. This included an evaluation of the related project management controls and a comparison of the information provided for fiscal 2003. We used control questionnaires and follow-up interviews with IT project managers to ascertain information about project management practices, as well as Agency infrastructure and architecture projects. Scope Limitations We could not substantiate how internal controls for EPA's Working Capital Fund process integrate with both the IT investment process and the Enterprise Architecture, despite repeated efforts to obtain relevant policy or procedural information from OEI officials. The Working Capital Fund is used to fund various aspects of IT projects. We were advised that responsibility for the Fund recently shifted from OEI to the Office of the Chief Financial Officer. The Working Capital Fund concept is described in the 42 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act narrative for the Agency's IT Architecture Roadmap, but the Roadmap does not elaborate on the Fund's relationship to the Agency's IT investment process. We attempted to audit two infrastructure project proposals: the National Centralized Computing and Information Processing Initiative and the proposal for the Scalable Computing and Information Infrastructure. The Agency could not provide any support for the proposals, including support for why $13 million in work included in initial proposals was no longer in the total costs of a subsequent proposal. Consequently, we could not audit what happened with the $13 million. Following our inquiries, the Scalable Computing and Information Infrastructure proposal was withdrawn from the investment review process and included as part of the National Centralized Computing proposal. Other projects also showed significant variability in Working Capital Fund expenditures, and we could not verily the nature of these variabilities. Congressional Concern One of the reasons for our conducting this review was the concern expressed by Congress in a report from the U.S. Senate's Governmental Affairs Committee, Investigative Report of Senator Fred Thompson on Federal Agency Compliance with the Clinger-Cohen Act, dated October 20, 2000. The report indicated that Federal agencies had not taken adequate actions to implement the Act, and noted that EPA did not produce evidence of any specific mission-related review of assessments based on programmatic or operational goals. EPA acknowledged shortcomings in its IT investment proposals, such as milestones being too general, projects being planned and managed in a stovepipe fashion, priorities not being established agency-wide, and the IRM strategic plan not being updated since the implementation of the Government Performance and Results Act. Further, when the Committee asked for a status report on EPA's top 10 IT investment projects, EPA could not provide any information on the status of 4 of those 10 projects. The Committee made numerous recommendations to executive departments (including EPA) for making improvements. Prior Audit Coverage In OIG Report No. 2001 -P-00013, Water Enforcement: State Enforcement of Clean Water Act Dischargers Can Be More Effective, dated August 14, 2001, we reported that although the modernized Permit Compliance System was estimated to cost more than $10 million in life cycle costs, the required system charter and system management plan decision papers had not been prepared or approved by appropriate levels of management. In OIG Report No. 001000239, Financial Management: EPA 's Fiscal 1998 Working Capital Fund Financial Statements, dated March 29, 2000, we found internal control weaknesses that would impact the overall management of Working Capital Fund operations, and resulted in managers not having accurate or timely financial information 43 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act on the Fund's operations. This Fund provides EPA with computer and telecommunication services on a cost-reimbursable basis. In OIG Report No. E1NMF3-15-0072-5100240, Management of Application Software Maintenance at EPA, dated March 31, 1995, we noted that while EPA was creating the Working Capital Fund to more cost effectively administer services, it was still questionable whether EPA could separate application software maintenance activity from operations activity. EPA did not develop, review, and update software maintenance costs by individual systems throughout their life cycles, which would prevent informed budget decisions from being made. In OIG Report No. El SKG3-15-0098-4400038, Special Review of EPA's Information Systems Program, dated March 24, 1994, we noted that management did not treat information as a strategic resource nor IRM as a core function and valuable tool. EPA did not have an information data architecture, data standards, or administrative structure to facilitate data sharing Agency-wide, and data quality problems existed. Also, a National Academy of Public Administration report, Transforming Environmental Protection for the 21st Century, dated November 2000, noted the nation needs authoritative information about environmental conditions, and discussed various steps being taken by EPA to do so. The report also emphasized that OEI had not begun to draft a strategic plan to guide its activities, and had no direct authority over the budget or staff that support EPA's systems. 44 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Appendix 2 Office of Environmental Information's Response to Draft Audit Report July 2, 2002 MEMORANDUM SUBJECT: Response to the Draft Report: EPA 's Management of Information Technology Resources Under the Clinger-Cohen Act, Audit Number 2001-0591 FROM: Kimberly T. Nelson /s/ Rick Otis for Assistant Administrator and Chief Information Officer TO: Nikki Tinsley Inspector General This memorandum provides a response to the Office of Inspector General (OIG) findings outlined in the Draft Report: EPA 's Management of Information Technology Resources Under the Clinger- Cohen Act, Audit Number 2001-0597, dated April 26, 2002. Overall, the Office of Environmental Information agrees with your emphasis on the critical importance of an effective IT resource investment management program that 1) delivers real benefits to the Agency's mission and 2) properly manages the risks across our enterprise portfolio. It is my intent to aggressively address the key issues raised in the report and I appreciate the work of your staff in providing us with this critical input to our planning and operation of the Clinger Cohen CIO program. We will provide a complete action plan for improvements upon receipt of the final report. There are some findings and recommendations in the draft report that my staff finds are not totally accurate in their characterization of the past accomplishments, current status and strategic directions of our program. We previously provided comments correcting some items which provided the basis for this draft report, but the report does not reflect any changes for those issues. We have also made much progress as an Agency during and following the audit. I would appreciate your review of our attached comments. Please adjust the final version of the report to incorporate changes to the introduction, findings and recommendations based on this information to ensure the final report provides the most accurate view of the program and where the Agency should focus attention and resources to help it improve in the future. 45 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act If you have any questions regarding this response please have your staff contact Mark Day, Director of the Office of Technology, Operations and Planning at (202)566-0300. Attachments cc: Mark Day, Director, Office of Technology Operations and Planning Debra Stouffer, Chief Technology Officer Kathy Petruccelli, Director, Office of Planning, Resources and Outreach Mike Flynn, Deputy Director, Office of Information Analysis and Access Brion Cook, Director, IT Policy and Planning Division Rick Martin, Director, National Technology Services Division Kevin Phelps, Associate Director, IT Policy and Planning Division Barbara A. Chancey, Chief, IT Strategic Planning Branch Chuck Cavanaugh, Program Lead for Investment Management John Sullivan, Chief Architect John Moses, Office of Information Collection Joe Dillon, Comptroller Juliette McNeil, Director, Financial Management Division John Gherardini, OAM Tom McEntegart, OAM Ed Lillis, OA Edward Cottrill, OW Tony Jover, OSWER Michael Mundel, OECA Jeffrey Worthington, OEI Audit Coordinator Brigid Rapp, OCFO Audit Coordinator Christa Eckel, OAM Audit Coordinator Greg Marion, OECA Audit Coordinator Judy Hecht, OW Audit Coordinator Johnsie Webster, OSWER Audit Coordinator Patricia H. Hill, OIG James Rothwell, OIG 46 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Draft Report: EPA's Management of Information Technology Resources Under the Clinger-Cohen Act, Audit Number 2001-0591 Executive Summary While we agree with the overall goal of the report, in many cases findings do not adequately reflect status and accomplishments, so recommendations are not as helpful as they might be. We request adjustments to findings and recommendations to focus attention more effectively on where additional effort and resources would benefit the Agency. The following comments address statements in the Executive Summary "Results in Brief' which contains content outlined from each chapter. Additional specific comments on findings and recommendations are identified separately in relation to the respective chapters. "Since established in 1998, EPA's CIO has not taken adequate actions to implement and institutionalize the Agency-wide authority and responsibilities for IT capital investments" EPA CIO's have made major advancements in ensuring Agency-wide compliance with Clinger-Cohen responsibilities. EPA established the Quality Information Council (QIC), chaired by the CIO and comprised of Agency senior resource management officials. The QIC formally approves IT investment decisions, and has done so since Clinger-Cohen has been in place. Under CIO's leadership, EPA senior resource managers have engaged in substantive investment reviews and direction. Their joint efforts have lead to restructuring of portfolio components, as well as substantive change/improvement of specific proposals. "Several key factors continue to inhibit the realization of a successful program..." OEI has made significant advances on each of the factors specified. Specifically the CIO has taken steps to: • establish a substantive range of new policies, procedures, and guidance on priority areas (security, investment) and is in the process of moving forward on a new comprehensive policy framework; • promulgate a new information strategic plan reflecting the Clinger-Cohen framework (in CIO review); • officially establish a chief architect and elevate the Agency profile for enterprise architecture development; • hire a Chief Technology Officer to champion Clinger-Cohen compliance within EPA • employed risk-based assessments for capital IT projects reflecting the evolving nature of OMB guidance under Clinger-Cohen; • establish new IT cost-tracking structures and requirements, and begin integrating investment and cost- tracking. "CIO had minimal assurance that IT investments reported to OMB would maximize their value" CIO recommendations for IT investments reflected senior Agency decisions on strategic program direction and value, based on then applicable Agency needs and available OMB guidance. Further, OEI continues to strengthen the investment review process to maximize value, including regular investment reviews of all OEI investments to review cost, schedule, and performance. 47 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Executive Summary Recommendations OEI/OTOP Response Revise outdated policies to remove unauthorized IT business practices and add new requirements. Suggest restating to acknowledge OEI process underway since Ql/02 to: 1. Identify, from a best practices perspective, what EPA's IT policy collection should be (recommendations to be forwarded for CIO review in August, 2002); 2. Catalog EPA's current IT policy collection (completion in August, 2002); 3. Identify the gaps between the "should be" and "current" states i.e., those IT policies needing to be created, updated, or canceled (September 2002); 4. Develop a multi-year plan for how to address the gaps and bring EPA's IT policy collection to the "should be" state referencing Enterprise Architecture, CPIC, and IT acquisition processes (November, 2002). Finalize the IRM Strategic Plan. Agreed and underway. A "Strategic Information Plan" document is in CIO review. The goals and direction put forth in this document are being incorporated as drivers in the architecture development. Formally establish a Chief Architect position with sufficient authority. Please correct. On February 22, 2002, the CIO established the Enterprise Architecture Program and named John Sullivan as Chief Architect for EPA. Implement an automated project management system (I-TIPS). Please restate: "Continue efforts to implement I-TIPS". OEI is implementing I- TIPS successfully and will be using it to generate OMB reports this September for budget year 2004. EPA completed a security vulnerability assessment and developed risk mitigation plans prior to production as required by OMB, and is now moving forward agressively. Implement monitoring and evaluation processes for IT investments. Please provide greater specificity. EPA senior management and the CIO do monitor and evaluate IT investments, reviewing all OEI investments for cost, schedule, and risk. Further, the CIO is taking steps to integrate investment, enterprise architecture, system life-cycle and fiduciary management processes in partnership with OCFO. A general statement expressing support for these efforts would be useful. Postpone funding for IT projects that have been identified as "materially deficient" The CIO and the QIC review investments prior to funding. Funding has never been recommended for an investment determined to be "materially deficient." 48 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Chapter 2 - CIO Needs to Fully Implement Clinger-Cohen Act Requirements. Findings Response 2.1 - Five years after implementation of the Act, EPA's CIO still had not established an adequate structure with the policies and guidance needed to sufficiently implement the Act. EPA through actions by the Administrator and the CIO has taken steps to implement critical Clinger-Cohen functions, and to direct IT resources in a manner that will deliver increasing value to our program mission. CIO leadership has been highly visible in enterprise architecture, investment management, critical policy, and workforce development. Please amend this finding to highlight the specific areas where the CIO and Agency leadership should direct additional attention and resources. 2.2 - Overall, EPA's program managers are treating the Clinger-Cohen Act requirements as little more than a paper exercise to satisfy the annual OMB budget call. Inaccurate. There is evidence that program offices do take the CCA seriously. Agency managers at multiple levels have actively participated in investment reviews. Management attention is reflected in: consolidation and elimination of duplicate projects; the number of program offices seeking OEI's consultation on preparing proposals; more refined reporting of budget numbers; linking IT investments to GPRA goals and agency priorities. It would be helpful, if you could expand the recommendation to identify the specific manner in which program managers should be involved beyond the roles that they currently fulfill (proposal preparation, approval, participation in Agency-wide portfolio development). 2.3 - Numerous examples demonstrated the use of inconsistent criteria and a general lack of objective, quantitative investment criteria (e.g., cost-benefit analysis) For the past five years, criteria has been based on the OMB's eight Raines Rules, plus additional Agency policy and programmatic criteria that was approved by the QIC's Information Investment Subcommittee (IIS), CIO, CFO, and the QIC, as such was both consistent and objective. This year, we plan to revisit selection criteria and approved revised criteria (including applying weights) through the QIC. 2.4 - EPA has not formally appointed a Chief Architect to oversee the development and execution of its Enterprise Architecture Plan. Inaccurate, please remove. On February 22, 2002, the CIO has appointed a Chief Architect for EPA. The Enterprise Architecture baseline, target and sequencing approach is scheduled to be delivered to OMB on October 15, 2002. 2.5 - The fiscal 2002 budget did not identify an architecture project. Inaccurate, please remove. For the fiscal 2002 budget, the architecture project was included as a component of integration proposals and for F Y02 it was reported separately on the Exhibit 53 - Section 3.- Architecture. 2.6 - In 2001, EPA purchased a SLA to use the off-the-shelf software I-TIPS.... However, when requested, EPA could not provide any evidence to support that they were assigning resources or providing milestones for implementing the software. Please restate. The Investment Management Team has assigned resources to I- TIPS implementation, developed milestones for production, proceeded with implementation, and will be using I-TIPS to generate automated OMB reports for this investment cycle. Further, I-TIPS will be expanded agency-wide in 2003. 49 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Chapter 2 - CIO Needs to Fully Implement Clinger-Cohen Act Requirements. Findings Response 2.7 - In addition, the following effects are likely to occur: 1) IT investments will no be driven by business priorities and mission goals; 2) Stovepipe systems will continue to operate; 3) EPA will continue to invest in duplicate IT system; 4) IT investments will not take advantage of technology advances and reduced costs; 5) reporting processes will not be made efficient for states and private industry; 6) application systems will not comply with environmental data and interoperability standards; and 7) increased public access and security requirements will not be met. Please restate to acknowledge the following: All IT investments in the CPIC process are linked to the Agency's strategic goals. Significant reductions in stovepipe systems have been made through consolidation and / or modernization to align these systems to the architecture. Duplicate systems have been identified through the CPIC process by the technical and executive management review. Proposals (e.g., Records and Document Management, and GEO and GIS) were combined last fiscal year to reduce redundancies and maximize efficiencies. For the past four years, data standards questions have been required, evaluation criteria has been established, and a data standards team has reviewed proposals to ensure that programs are complying with data standard requirements. Over the next couple years, Central Data Exchange (CDX) will be implemented. As CDX grows and gains wider acceptance, it will reduce the reporting burden on the states and private industry. Also, as the National Environmental Information Network is being constructed with input from the states and industry. The new network will greatly enhance the reporting and information exchange between the states, industry, tribes and the agency. 2.8 - During recent years, the CIO should have used an IT investment control process to solve key Agency- wide problems such as integration of environmental data, electronic reporting,, duplicate systems, Geospatial Information, and data management. Please restate to acknowledge those very issues targeted and addressed during the CPIC process. For example, the GEO investment was stopped from receiving operating plan funds in F Y01 due to CPIC process findings. Also, duplicate systems were identified and requested to coordinate development strategies and present before the IIS. Recommendations Response 2.1 - Assign sufficient resources and expertise to ensure timely and effective implementation of report recommendations; and use objective, risk-based criteria to decide whether proposed and ongoing IT investments will help resolve key Agency-wide problems and advance EPA's IRM vision. Agreed. Request for an increase in resources (extramural, FTE) has been submitted for the F Y03 and F Y04 budgeting years. From the inception of this process under Clinger-Cohen, management reviews have been risk-based. The initial method referenced the "Raines rules" following the approach which was then applicable on a government-wide basis. In the FY 02 CPIC process, the technical review team is using objective, risk- based criteria by identifying weaknesses and working with program offices in producing strong business cases, cost/benefit analysis, results-oriented performance measures, cost and schedules, and presenting proposal evaluation results to the QIC/Information Investment Subcommittee in a portfolio management enterprise perspective. Increased resources would enable more frequent investment reviews. 50 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Chapter 2 - CIO Needs to Fully Implement Clinger-Cohen Act Requirements. Findings Response 2.2 - Revise EPA Directive 2100 and related guidance to remove outdated and unauthorized IT business practices. Incorporate appropriate policies and procedures for the Enterprise Architecture, CPIC process, and IT acquisitions addressed in the Clinger-Cohen Act, OMB guidance, and EPA Delegation 1-84. Please restate to acknowledge that OEI has had a process underway since Ql/02, anticipated for completion in Q2/03. The process is to: 1. Identify, from a best practices perspective, what EPA's IT policy collection should be (recommendations to be forwarded for CIO review in August, 2002); 2. Catalog EPA's current IT policy collection (completion in August, 2002); 3. Identify the gaps between the "should be" and "current" states i.e., those IT policies needing to be created, updated, or canceled (September 2002); 4. Develop a multi-year plan for how to address the gaps and bring EPA's IT policy collection to the "should be" state referencing Enterprise Architecture, CPIC, and IT acquisition processes (November, 2002). 2.3 - Work with the Director for Acquisition Management to (a) direct contracting officers and other procurement personnel to only accept procurement requests with a formal CIO approval or officially re-delegated procurement authority; and (b) establish interim delegations, policies and procedures for IT procurement, until formal redelegations are revised and implemented. Please restate to acknowledge that OEI (and previously OIRM) has historically worked with OARM to ensure appropriate re view/concurrence for IT acquisitions. The CIO has initiated the establishment of delegations under the Clinger-Cohen framework to ensure all IT procurements have formal management official approval (either CIO or someone with formal authority delegated by the CIO) before consideration by procurement personnel. Chapter 3 - Weaknesses in CPIC Process Place EPA's IT Investments at Risk Findings Response 3.1- However, EPA's CPIC process does not monitor each project's execution during a Control phase nor evaluate the adequacy of completed projects in an Evaluation phase, as recommended in Figure 2. Please restate. EPA's CPIC process has incorporated the Control phase since its inception. The evaluation phase is currently being implemented. Refer to Report on Management Options for Implementing the Evaluation Phase of IT Capital Planning and Control, dated January 7, 2001 and white paper entitled Implementing the Select/Control/Evaluate Phases of Review, dated April 12, 2001. To be helpful, please identify, in the final report, specific aspects of control / evaluate phases which OIG believes require further attention. 51 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Chapter 3 - Weaknesses in CPIC Process Place EPA's IT Investments at Risk Findings Response 3.2 - The peer review risk assessment was the only substantive process used to control IT investments, and we found no evidence of a quality assurance process to ensure investment proposals were accurate. This finding is inaccurate as stated. The CPIC process has four levels of qualitative reviews: 1) staff level - a thorough review of proposal format and content is conducted; 2) technical peer review - evaluation criteria based on the Raines Rules is applied and proposals are evaluated based on technical merit, then grouped and ranked; and 3) IIS - executive management level review to address funding and policy issues, grouped and ranked red, yellow or green. 4)QIC review. 3.3 - Agency management planning and budgeting recommendations for fiscal 2002 were based on IIS opinion, rather than objective peer review risk evaluations. This finding is inaccurate as stated. The FY 2002 recommendations were based on the technical peer review analysis and the discussions and deliberations of the IIS. The IIS depends heavily on the technical review results. 3.4 - Table: EPA Major Investment Proposals, Key Project Risk Factors (Fiscal 2002) The percentages in these findings do not match reports and OMB statements given by EPA and OMB. It would be helpful if the OIG presents the document which states "OMB Assessment" amounts. Please also include the statement from OMB "We think a great deal of BCA has been performed on the majority of the portfolio." 3.5 - Nevertheless, the IIS recommended to the Quality & Information Council and the CIO that all 48 projects be recommended for funding in the fiscal 2002 budget submission. This finding is misleading and should be restated or removed. Projects were recommended for funding only after substantial corrective actions were taken to make the business case, and a fourth review of the project proposal was conducted. Five projects were required to address the IIS to explain and defend their business cases. 3.6 - Major projects were found to have material deficiencies by the peer review process, yet the IIS recommended to fund these projects in fiscal 2002. In fiscal 2003, the peer review process once again stated these projects contained significant weaknesses, but IIS still recommended them for funding. This is inaccurate. In 2002, the IIS red-lighted five projects, initially not flagged by the technical peer review team. These projects were required to go before the IIS for further scrutiny and extensive review of the project's business case occurred. The finding should also state that for 2003, following extensive project/portfolio revisions per senior management direction, OMB subsequently found deficiencies to the business case for only 2 of 48 proposals, which they then accepted after minor revisions. 3.8 - Paragraph on CIO Needs to Formalize and Institutionalize a CPIC Process - The CIO has yet to establish policies and guidance, and implement key Clinger-Cohen Act requirements by formalizing the CPIC process in Agency Directive 2100. The process is formalized, so please restate. For the past five years EPA has been conducting a Capital Planning and Investment Control process (CPIC), which includes a Select and Control phase, appropriate guidance, training, evaluation criteria based from the Raines Rules, a formal technical review process and executive management review to evaluate proposals. In June 2002, a final CPIC policy was issued, formalizing the process in Agency Directives. 52 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Chapter 3 - Weaknesses in CPIC Process Place EPA's IT Investments at Risk Findings Response 3.9 - Implementing the Federally sponsored I-TIPS software, an automated investment control and reporting system, would provide EPA with a valuable tool for monitoring and managing its IT investment portfolio. While EPA has been using a peer review process to evaluate risks, management has not employed a structured CPIC process to maximize the value of investments and manage the risks of IT acquisition projects. Please restate. There are two separate issues - implementation of I-TIPS and a structured CPIC process. OEI is implementing I-TIPS and will be using it to generate OMB reports for this cycle. OIG should also note that EPA's schedule for I-TIPS implementation reflects the fact that I-TIPS does not conform to the Agency's existing technical architecture and employs web-based functions with security vulnerabilities which required careful risk assessment and mitigation plans, prior to production. OEI has developed methods to address vulnerabilities and is moving forward. From a process perspective, EPA has consistently followed a highly structured approach involving project and program managers at key decision points. The process continues to evolve and next year will integrate enterprise architecture with investment to provide further structure to the process of establishing management priorities and decision making. If OIG believes additional structure is required, specific recommendations would be helpful. Chapter 3 - Weaknesses in CPIC Process Place EPA's IT Investments at Risk Recommendations Response 3.2 - Formally establish objective, risk- based criteria for the IIS to use in selecting and funding all IT investments (e.g. Chart of EPA System Development Risk Factors). Based on the criteria, management should not fund proposals or projects that classify as high risks. Please restate. EPA has in fact employed specific evaluation criteria for review of CPIC proposals for past CPIC cycles. The evaluation criteria was released as part the annual Exhibit 300 data call so that preparers and reviewers were aware of the criteria each proposal would be evaluated against. Also, the IIS will be given a technical peer review summary of each of the proposals prepared in accordance with the evaluation criteria and with that information will be following the OMB scoring guidelines provided in this year's A-l 1 guidance. As part of the Strategic Direction for Investment Management, the IIS plans to identify business and architectural criteria for investments, Ql/03. With this established, the Agency will be able to make even more thorough, objective, risk-based evaluations of all proposals than in the past. Additional specific suggestions from OIG on how to further enhance criteria would be welcome. 53 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Chapter 3 - Weaknesses in CPIC Process Place EPA's IT Investments at Risk Recommendations Response 3.3 - Postpone funding for current IT projects identified by the Peer Review process as materially deficient or high risk for 2 consecutive years, until critical deficiencies are resolved and the resolution steps adequately documented. Agreed. However, no "materially deficient" project has been recommended for investment by the CIO. The Agency's Information Investment Subcommittee is responsible for recommending funding to the CIO regarding major IT investments. Those projects identified in the Technical Peer Review process as deficient are afforded the opportunity to make revisions to their proposals prior to the Subcommittee's review and, time permitting prior to the QIC's review. Forty-eight proposals submitted for funding last year to OMB were approved. For example, in FY01, the IIS advised the Acting CIO to charge a task force to develop a strategic direction and architecture for electronic records, dockets, and document management applications. The IIS made a recommendation to suspend funding for seven systems. The Acting CIO followed through on that recommendation (memo from Margaret Schneider, dated October 12, 2001, Management Task Force for Agency Document Management Systems, "...suspend spending for design and development work for all new and existing document management systems.") 3.4 - Direct the Information Investment Subcommittee to monitor the execution of IT projects during the fiscal year (at least quarterly) to identify emerging cost or schedule problems and initiate corrective actions. Agreed. As part of ITPPD's Investment Management strategic planning efforts, and in conjunction with the use of I-TIPS, it is OEI's vision to evolve the Agency's capital planning process to do a continuos update and review process in the next two years. This continuos process will involve Program Offices updating their business cases as their systems develop (i.e. moving from different life cycle or CPIC phases). Rather than relying on annual data calls for updates, this will allow the Program Office management, the Subcommittee, the QIC and the CIO's office access to the most current information possible, thus providing them the ability to address cost or performance issues as they are identified, not just once a year. 3.5 - Initiate a formal process with written evaluations of ongoing, completed, and terminated information technology projects to evaluate whether the projects or systems are successfully delivering promised benefits at an acceptable cost. Agreed. As ITPPD prepares its Strategic Planning for IT Investment Management, one of the areas being examined is the formalization of processes and evaluations in all phases of the CPIC process, including the possible inclusion of Pre-Select and Steady State phases to provide management with on- going evaluation monitoring. 3.6 - Implement an automated project management system (e.g., I-TIPS) to provide timely, reliable information for investment decisions. This recommendation should clearly define the difference between a portfolio management system and a project tracking system - I-TIPS is a portfolio management system, not a project management system. ITPPD is currently piloting the use of I-TIPS in EPA. ITPPD plans to populate Exhibit 300 data in I-TIPS and submit electrically to OMB (09/02). Additionally, ITPPD plans to develop an Agency-wide deployment strategy for I-TIPS in Ql/03 for FY 03-04 implementation. 54 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Chapter 4 - EPA Needs to Organize and Integrate Planning for IT Investments Findings Response 4.1 - During 2001, EPA completed many actions towards establishing a baseline enterprise architecture for IT planning purposes. In April, EPA provided OMB with documentation of EPA's first Enterprise Architecture, dated March 29, 2001. However, by October, neither the Agency's IT Contracting Officer nor the Procurement Office had been provided a copy of the proposed Enterprise Architecture. This is incorrect. The Agency's Enterprise Architecture is posted on the EPA Intranet and program offices were notified of its availability. The Office of Administration and Resources Management (OARM) was notified that the architecture had been published. 4.2 - Moreover, the document was not provided timely to the EPA program offices for use in developing IT investment proposals for the fiscal 2003 budget submission. Please restate. This finding does not accurately reflect that appropriate guidance was provided from the EA Team to proposal preparers on developing their 2003 / 2004 investments. The EA Team also worked one-on-one with program offices requesting assistance. The current enterprise architecture being developed will contain a baseline, target and sequencing approach, which will assist preparer in the 2005 exercise. 4.3 - Also, OMB reviewed the Agency's fiscal 2001 IT Investment Portfolio and noted that they could not match the projects in the proposed Enterprise Architecture to the portfolio. In August 2001, OEI established a workgroup to identify and verily EPA's business processes for the Enterprise Architecture baseline. The work group's efforts occurred after completion of our field work; as such, we do not know fully what they have accomplished. Please acknowledge that the workgroup has updated the business processes and these processes will be aligned with the new OMB Business Reference Model. 4.4 - EPA's outdated IRM Strategic Plan has contributed to the delay in implementing the Enterprise Architecture concept. In May 2001, EPA established an agency-wide work group to update the IRM Strategic Plan. The work group provided the draft plan to OEI's Quality Information Council, but it has yet to be finalized. Please restate to acknowledge that a "Strategic Information Plan" document is in CIO review. The goals and direction put forth in this document are being incorporated as drivers in the target architecture development. 55 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Chapter 4 - EPA Needs to Organize and Integrate Planning for IT Investments Findings Response 4.5 - As of the end of field work, EPA had yet to fully baseline and validate the Agency's business processes essential for establishing a portfolio for future IT investments. EPA's draft Enterprise Architecture document included very high-level business processes; however, these processes had yet to be validated by the responsible program offices. We were informed that some of these business processes have been revised, but were unable to substantiate whether the applicable program offices formally endorsed the work group's conclusions. Agreed, however we have made progress, and plan to acquire QIC approval of the EA. Formal validation of baseline program components by the CIO and senior program managers, via the QIC, is occurring this year per the management plan for build-out of the enterprise architecture. 4.6 - Therefore, the physical and personnel security requirements of these servers need to be added into the baseline security architecture. Inaccurate. The Security Architecture does address the physical, facility and personnel security issues. 4.7 - As depicted in Figure 3, the Enterprise Architecture conceptual framework should consist of five components. As such, the Enterprise Architecture should define mission- critical data needs to properly support the IT investment process. However, EPA's current Enterprise Architecture does not adequately address (1) EPA's existing data standards and related metadata baseline information, and (2) other critical data used by stakeholders and programs business processes. This finding is inaccurate. Data standards and critical data are both integral aspects of EPA's enterprise architecture. The model specifically references data standards, and will address program-specific data needs across several dimensions. 4.8 - We had recommended that EPA support its data standards program by using the Environmental Data Registry as a central repository for publishing and recording data standards. EPA has yet to do so, and the draft Enterprise Architecture does not adequately describe the registry as a critical component of its target architecture. This finding is outdated and should be removed. The Enterprise Architecture does support data standards and the EDR. The document being prepared for OMB will clearly outline this architectural component. 56 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Chapter 4 - EPA Needs to Organize and Integrate Planning for IT Investments Findings Response 4.9 - Complete Inventory of Systems Neededfor Enterprise Architecture ... we found that the Enterprise Architecture document does not include sufficient information on Agency application interfaces. The document states the CIO plans to gather and document this information as part of the Agency's ongoing application inventory initiative, including documentation regarding major interfaces with applications outside of the Agency. The Enterprise Architecture will gather more information this year on internal system interfaces and partner interfaces within the framework of the National Environmental Information Network (NEIN). The target architecture and sequencing plan will also take into account the impact of external federal Agency interfaces and E-gov directions. 4.10- Our review showed that the Enterprise Architecture document did not explicitly identify minimum response times for key transaction- based systems and for business application systems on the Agency's wide area network. This finding does not provide relevant or helpful direction. Normally, this level of detail is not in an Enterprise Architecture document. Transaction response requirements for critical data streams will be considered as a factor in the development of the technical architecture, which must be scaled and engineered to support such needs. 4.11- We believe the VPN concept is needed today to help the Agency comply with existing Federal telecommuting statutory requirements and to satisfy current business needs. OEI agrees with the importance of secure external communications. This year OEI is taking the critical steps to establish secure external partner levels of access with implementation planned to start next year and full operations to be available on an enterprise basis in 2004 (pending continued availability of resources). 4.12 - Also, EPA needs to define the role and authority of its Chief Architect for IRM. The role of this Chief Architect is to oversee development and coordination of the Enterprise Architecture with other planning elements that should materially shape and drive the IT planning structure. The CIO named an individual to this role in February 2002 (via electronic mail), but there has been no formal definition of the position's scope and responsibilities, nor any official delegation of authority. This recommendation is outdated and should be refined. On February 22, 2002, the CIO via electronic email, established the Enterprise Architecture Program and named John Sullivan as Chief Architect for EPA. If additional authority is needed in the view of OIG, specific deficiencies should be noted. 57 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Chapter 4 - EPA Needs to Organize and Integrate Planning for IT Investments Findings Response 4.13 - To EPA's credit, management established a central Enterprise Architecture workgroup in August 2001. However, no permanent central organization has been established or assigned resources to coordinate, develop, and maintain the Enterprise Architecture. Agency-wide Enterprise Architecture components need to be addressed and maintained for the following functional areas: the identification of EPA's major and significant systems; defining the security architecture; validating the business processes with program offices; developing the Middleware architecture and defining baseline telecommunication requirements; defining Working Capital Fund capital investments; and approving individual IT project management plans for major projects or systems. This finding should be rephrased. The functional areas identified in the recommendation are all included within the strategic activities underway this year and planned for next year. Please restate the recommendation to acknowledge the importance of the ongoing efforts being made to address these needs. 4.14 - In its fiscal 2003 budget submission, EPA identified the Information Integration Program as its only major architectural project for deriving and completing an enterprise architecture. As critical as the project is to EPA's Enterprise Architecture development efforts, no final management work plan has been implemented for this project since the draft was issued in December 2000. A final work plan is essential to ensuring the timely success of the individual program, as well as the overall quality of the Enterprise Architecture Plan and the Agency's future technology investments. Although EPA views the program as key to improving the overall integration of environmental information, this project does not report to the Chief Architect. This finding is inaccurate and does not accurately reflect the continuity of results and the connection of that project with the Enterprise Architecture program. The products from the Information Integration Program are the basis for the target architecture of the environmental business area. Please restate this finding to acknowledge the intent and proposed products of the Information Integration Program. 58 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Chapter 4 - EPA Needs to Organize and Integrate Planning for IT Investments Findings Response 4.15- The Chief Architect provided information that indicates EPA's program and regional offices will be asked to co-develop the Agency's baseline and target elements for the enterprise architecture. With OEI's leadership and facilitation, the program and regional offices will conduct their own architectural needs analysis, and realign their respective systems with EPA's evolving target. We were unable to substantiate how this will be accomplished. The participants will need a clear understanding of their roles and responsibilities, as well as their respective business processes, if they are to play a significant role in helping define the enterprise architecture. We agree that clear roles and responsibilities are essential in defining the Enterprise Architecture. We have taken the necessary steps to ensure participants are clearly aware of their respective roles and responsibilities. The Chief Architect and the Enterprise Architecture team are working with program and regional representatives at the staff level to develop requirements and validate Agency-wide perspectives. At the same time, the Chief Architect is preparing explicit guidance, including senior management roles, to formally record roles and responsibilities of program and regional offices for the architecture. This framework for Enterprise Architecture policy and practice will be reviewed by the CIO and senior managers at a forthcoming QIC meeting in July, per the schedule presented to the QIC on 6/26/2002. Chapter 4 - EPA Needs to Organize and Integrate Planning for IT Investments Recommendations Response 4.1 - As the number one priority, we recommend that the Chief Information Officer formally establish: (a) an Enterprise Architecture program to plan, manage, monitor, and control the development and maintenance of the plan. (b) the Chief Architect position by clearly defining the role, responsibility and authority of the job. The position should ensure a system of accountability for the overall architectural effort. This would include coordinating and overseeing resources for IRM strategic planning and the Information Integration Program, and reporting directly to the CIO. This recommendation should be rephrased to acknowledge the efforts underway to plan, manage, monitor and control the development and implementation of the Enterprise Architecture. The Chief Architect, through direct and ongoing consultation with the CIO, has been directing and coordinating the Agency' s efforts to create an architecture and architecture program. The Chief Architect is working with the CIO and Chief Technology Officer (CTO) to promulgate an Agency-wide framework for managing the establishment and implementation of the Enterprise Architecture. This framework will be a major focus for senior executive discussion and decision at the July meeting of the QIC. We would appreciate any subsequent OIG recommendations that focus on additional steps required to support this effort. 59 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Chapter 4 - EPA Needs to Organize and Integrate Planning for IT Investments Recommendations Response 4.2 - Under the leadership of the Chief Architect, update and maintain the Agency IRM Strategic Plan to support EPA's Strategic Plan, its Government Performance and Results Act requirements, and the Enterprise Architecture. The Chief Architect and the Architecture Team are responsible for creating, updating and maintaining the Agency's architecture. As part of creating the architecture, the Chief Architecture must coordinate and participate in the strategic planning process, GPRA and other efforts. OEI is producing a "Strategic Information Plan" under direction of the OEI - Office of Information Collection (OIC). This Plan will be used as a driver for the EA development. 4.3 - Identify current major and significant general and application systems to establish an accurate inventory of such systems and integrate this information with both the Agency's Enterprise Architecture application component and the IT CPIC Portfolio. EPA agrees with this recommendation. The Enterprise Architecture will incorporate all systems (major and significant as defined in the CPIC) and others into the Information Resources Registry System - which will serve as the Agency Applications Inventory. The IRRS is scheduled to be operational by the end F Y02. A linkage between the IRRS and the EA repository is planned. All application systems within the purview of CPIC review are included in the baseline applications architecture. 4.4 - Develop a master project plan for completion of all parts of the Enterprise Architecture, including a breakdown of the tasks and subtasks needed to acquire, develop, and maintain the Enterprise Architecture. EPA agrees with this recommendation. The Enterprise Team has an overall management plan and project plan that contains the detail tasks and subtasks to develop the Enterprise Architecture. Additionally, the Team is in the process of identifying a change management process for updates to the Agency's architecture. 4.5 - Establish an information repository, require the use of a data registry for Agency maintained data, map EPA's data and information resources, and adopt life-cycle data management principles for the Enterprise Architecture data and systems components. Please restate this recommendation to reflect efforts already underway. OEI has established an EA repository in which the Agency's business, data, applications, and technologies are mapped and interlinked. As part of the CPIC process, programs will be required to ensure their systems are represented in the EA repository and applications inventory. The Enterprise Architecture Team is coordinating efforts with the ITPPD's efforts to update the Agency's life-cycle principles currently being developed to produce a "cook book" on systems development that will align the Systems Lifecycle policy, the CPIC Process and the Enterprise Architecture. 4. 6 - Use a top management verification, validation, and approval process to ensure program business processes and goals are accurately reflected and incorporated into the Enterprise Architecture. Subsequently, formalize the process as a discipline for updating the Enterprise Architecture document. EPA agrees with this recommendation. The Enterprise Architecture is presented to the Quality Information Council for recommendation to the CIO for approval. An EA change management and configuration control process is being developed to formalize the process of updating the architecture. The Chief Architect is preparing explicit guidance, including senior management roles, to formally record roles and responsibilities of program and regional offices for the architecture. This framework for Enterprise Architecture policy and practice will be reviewed by the CIO and senior managers at a forthcoming QIC meeting in July, per the schedule presented to the QIC on 6/26/2002. 60 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Chapter 4 - EPA Needs to Organize and Integrate Planning for IT Investments Recommendations Response 4.7 - Coordinate the Enterprise Architecture document with the Agency's Office of Acquisition Management for future IT acquisitions. Jointly develop an approval process that ensures the Enterprise Architecture concept is incorporated in IT contract activities for large and significant IT projects. EPA agrees with this recommendation. In addition to formal promulgation of acquisition authority and delegations by the CIO, once the EA version 1.0 is approved by the CIO, the EA team will work with OAM to broaden the current contracting clauses to ensure compliance with the EA. 4.8 - Develop a Middleware Architecture as part of the Enterprise Architecture technology component to: define the components that interface among the client and server systems; improve the overall usability of the distributed architecture; and integrate the information repository with the client-server systems. Please rephrase this recommendation. As part of the Target Architecture (Q4/02), the data warehouse methodology and platforms will be determined. The detailed design of the warehouse (whether it is virtual or physical) will be contained in the Technical Reference Model, which is being developed as part of the EA. OIG recommendations should be cautious when making specific technical references (e.g. linking client-server systems with the repository) as the target technical architecture is likely to move the Agency towards new models. 4.9 - Establish a comprehensive and explicitly defined set of baseline telecommunications requirements to support a scalable, reliable, and secure network infrastructure for the Enterprise Architecture technology component. Also, address existing bandwidth shortages and provide for additional network capacity to support current business needs and take advantage of technology advances. OEI agrees with the importance of this recommendation and its importance for the technical architecture. Telecommunications requirements to support a scalable, reliable, and secure network infrastructure, bandwidth capacity, and additional network capacity are essential components of the Technology Architecture Segment. OEI is working with OCFO and senior agency managers to define a fiduciary and technical management strategy that will address current technical architecture shortfalls and provide more effective methods to maintain the technology in the future. Chapter 5 - EPA Needs to Strengthen IT Project Management Criteria Findings Response 5.1- Paragraph on No Reliance or Value Placed on EPA's IT CPIC Process Please discard this finding, it is inaccurate. Over the past five years of the CPIC process, the Investment Management Team has worked with over 50 different program mangers at one time or another. We have received positive comments from program managers that the process has forced them to rethink their investments and to pay closer attention to costs, schedule, and milestones. EPA does acknowledge and place value on the need for the IT CPIC process. 61 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act 5.2 - EPA had not adopted standard tools to help managers plan, control, and evaluate IT investment projects and track project costs, schedules, and resources. Please revise this statement, it is inaccurate. First, with the development of the CFO Comptroller Policy Announcement 01-10 and the IT Cost Tracking system, program offices are required to track project costs. Secondly, as program offices implement this requirement, it clearly complements and links to project planning and work plan development. 5.3 - The absence of key decision documents and senior management approval increase the risk that funded IT projects will evolve in an unstructured, untimely, and costly manner. Please restate this finding to acknowledge efforts of the senior management and decision making body of the QIC. The QIC, referencing recorded recommendations from the IIS, formally acts on each IT investment. Formal meeting notes are taken at each subcommittee meeting, reviewed and approved by the co-chairs, and starting in January 2002, co-chairs signed the meeting notes before being distributed to subcommittee members. Chapter 5 - EPA Needs to Strengthen IT Project Management Criteria Recommendations Response 5.1 - We recommend the Chief Information Officer monitor IT investments to ensure that SMPs are prepared in accordance with Agency requirements, and that they appropriately link to the respective Clinger-Cohen Act submission documents the Enterprise Architecture and other planning documents. EPA agrees with this recommendation. As OEI prepares its Strategic Planning for IT Investment Management, one of the areas being examined is the formalization of processes and evaluations in all phases of the CPIC process, including the possible inclusion of a Pre-Select phase. The Pre-Select phase will allow the Agency to ensure that all proposed systems in the system lifecycle planning process are aligned with Agency requirements on enterprise architecture, security, etc. This Pre-Select phase will allow EPA to ensure compliance with Systems Lifecycle Policy in advance of a system entering the Select Phase. 5.2 - We recommend the Chief Information Officer re-evaluate funding for IT investments that do not provide sufficient written justifications for projects exceeding budgeted costs or project milestone schedules by more than 10 percent. Please rephrase this recommendation to accurately reflect the current process in place. Systems without sufficient justification to cost and schedule variances greater than 10% are not recommended for funding. As part of the Exhibit 300 submission, OMB is requiring that all major systems provide a breakdown of costs and schedule performance from their original baseline. The Chief Information Officer does not recommend investments for projects with insufficient justification or those with excessive cost and schedule variances. 5.3 - We recommend the Chief Information Officer prescribe standard tools for managing system development projects and for managing software changes, as part of the development of consistent definitions of system life cycle stages to be used for IT systems and project management. The selected tool should be approved by the Chief Financial Officer as being compatible with the Agency's cost accounting system. OEI does not agree that it is necessary or appropriate to prescribe uniform tools for managing system development projects and software changes since it is unclear at this time that there is one set of tools which meets the needs of all system development efforts in a cost-effective manner. However, OEI does intend to broaden the scope and usefulness of I-TIPS with particular attention to linkages between I-TIPS and Agency financial data for IT cost tracking. OEI is also leading an effort to update EPA's System Life Cycle Policy. The updated policy will provide appropriate consistent definitions, lay out the requirements that must be met when an Agency office develops a new system, provide appropriate system development management methodology options, and encourage the use of "best practice" project management principles and techniques. The selected "tools" will be compatible with the Agency's Financial systems. 62 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Chapter 5 - EPA Needs to Strengthen IT Project Management Criteria Recommendations Response 5.4 - We recommend the Air Quality System Project Manager update the SMP for the Air Quality System project and obtain the signature of approval of the Assistant Administrator for Air and Radiation at the conclusion of the analysis stage and for major and significance enhancements. Please see the memo from William T. Harnett to Patricia H. Hill dated 5/28/02. 5.5 - We recommend the RCRA Information Project Manager revise the Project Management Plan for the RCRAInfo project to make it equivalent to an SMP, and update the document for planned system design changes and enhancements. In addition, the revised SMP should be formally approved by the Assistant Administrator for Solid Waste and Emergency Response to authorize funding for the IT investment and to ensure a system of accountability. Please see the memo from Marianne Lamont Horinko to Kimberly Nelson dated 6/14/02. 5.6 - We recommend the SDWIS/STATE Project Manager establish an SMP for the SDWIS/STATE project and obtain the signature of approval from the Assistant Administrator for Water at the conclusion of the analysis stage and for major and significant enhancements. We agree with this recommendation and SDWIS/STATE has all the components of a Systems Management Plan. However, the project has not compiled the information into a single document for signature for the following reasons: First, we have not been able to identify the format the agency wishes for the SMP and second, a SMP was not specifically required when the project began. Part of our plan for this fiscal year (may slide to early next FY) is to compile the document and present it to management. 5.7 - We recommend the Project Managers for the Air Quality System, RCRAInfo, and SDWIS/STATE link the SMP to the Agency Clinger-Cohen Act submission documents and the Enterprise Architecture and planning documents. OAR — Please see the memo from William T. Harnett to Patricia H. Hill dated 5/28/02. OW — When the SMP document is completed it shall be linked to all IT submissions (where applicable). OSWER - Please see the memo from Marianne Lamont Horinko to Kimberly Nelson dated 6/14/02. 63 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Chapter 5 - EPA Needs to Strengthen IT Project Management Criteria Recommendations Response 5.8 - We recommend the Project Managers for the Air Quality System, RCRAInfo, and SDWIS/STATE manage project development efforts in accordance with the SMP, as updated, throughout the life cycle of the system, and retain the SMP for reference and review by the CIO or the CIO's designated review official. OAR — Please see the memo from William T. Harnett to Patricia H. Hill dated 5/28/02. OW -- We agree that the documents that go into the SMP should be updated throughout the life-cycle of the system. We currently do this and with each new release the following documents are updated (among others): requirements, design, testing, and user documentation. Also, each fiscal year we produce a new work plan. Finally, we continuously, update and track our financial reports. OSWER - Please see the memo from Marianne Lamont Horinko to Kimberly Nelson dated 6/14/02. Chapter 6 - Project Cost Accounting System Vital for Planning & Managing IT Investments Findings Response 6.1 - Our concern is compounded by the fact that the three system life cycle categories set forth in the Policy Announcement are inconsistent with the phases described in EPA Directive 2100. Please restate this finding to accurately reflect efforts in the Systems Life Cycle work group and the IT Cost Tracking work group. Participants from OEI and OCFO are on both work groups coordinating the IT Cost Tracking system guidance, which includes policy development, and the Systems Life Cycle development, updating our system life cycle policy. The life cycle categories stated in the policy announcement reflect the new work that is being done to update the systems life cycle policy. Recommendations Response 6.1 - We recommend the Chief Information Officer, Chief Financial Officer, and Assistant Administrator for Acquisition Management work together to develop consistent definitions of systems life cycle stages and IT costs to be used for contracting, accounting, IT systems, project management, and the capital planning investment control process. Please acknowledge the current ongoing efforts underway to meet this recommendation. ITPPD is currently leading an effort to update EPA's System Life Cycle Policy. This effort will develop consistent definitions that can be used, to the extent practicable, throughout the Agency's varied processes that relate to IT systems development. Additionally, ITPPD is supporting OCFO efforts in developing an IT Cost Tracking system. As this system matures and focuses on capturing "actual" budget cost data more accurately, and comprehensive training is provided to program offices, management will be able to make better decisions to evaluate investment priorities. OCFO and OARM - submitting response under separate cover. 64 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Recommendations Response 6.2 - We recommend the Chief Information Officer, Chief Financial Officer, and Assistant Administrator for Acquisition Management work together to amend all current Agency software development contracts, and require that all future IT software development contracts be written to require a contractor to break out and separately report all IT software development costs by the system development life cycle. OEI - With the following ongoing efforts - the updated Systems Life Cycle Policy, the interim CPIC Policy (final soon to be released), architecture and the IT Cost Tracking system - the modular contracting approach will be supported, contractors will have better guidance on providing development costs, and management will be able to make better decisions on investments. Please acknowledge these efforts in your recommendation. OCFO and OARM - submitting response under separate cover. 6.3 - We recommend the CIO and Chief Financial Officer work together to develop consistent systems life cycle and IT costs definitions for revising EPA Directive 2100, and the interim IT activities policy guidance. OEI — Please restate this recommendation to accurately reflect the current efforts being developed between OEI and OCFO. ITPPD is currently leading an effort to update EPA's System Life Cycle Policy. This effort will develop consistent definitions that can be used, to the extent practicable, throughout the Agency's varied processes that relate to IT systems development. OCFO - submitting response under separate cover. 6.4 - We recommend Chief Financial Officer lead an effort to complete a needs and feasibility assessment of alternatives to determine what types of project cost information and supporting documentation are needed for the capital planning investment control process and managing IT projects. Submitting response under separate cover. 65 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act 66 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Appendix 3 Office of Chief Financial Officer's Response to Draft Audit Report July 19, 2002 MEMORANDUM SUBJECT: Draft Report on Management of Information Technology Resources Inspector General Audit Number 2001-0591 FROM: Joseph L. Dillon /s/ Comptroller TO: Patricia Hill Director for Business Systems (2421) I appreciate the opportunity to respond to your draft report titled "EPA's Management of Information Technology Resources under the Clinger-Cohen Act," Audit Number 2001-0591. The Office of the Chief Financial Officer (OCFO) fully supports your emphasis on effective management controls over EPA's information technology (IT) portfolio and, as you recommend, we are working closely with the Office of Environmental Information (OEI), the Office of Administration and Resources Management (OARM), and others. Chapter 6 of your draft, "Project Cost Accounting System Vital for Planning and Managing IT Investments" makes four recommendations for OCFO. A discussion of recent OCFO progress in implementing IT cost accounting is below. Specific responses to your draft recommendations for OCFO are attached. As you note, Comptroller Policy Announcement No. 01-10, "New Information Technology Accounting Requirements" (PA), has been in effect since October 1, 2001. The PA established a standard method of tracking all IT related costs in the Integrated Financial Management System (IFMS). As the PA states, OCFO recognizes that the IT cost accounting "procedures represent a new way of doing business in the Agency." We are now evaluating results and have implemented a quality assurance process to ensure the accuracy of the cost data for both large IT systems and projects, and for smaller projects and general IT activities. 67 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act To help familiarize staff with the new information and its uses, an IT Cost Accounting section has been added to OCFO@work at http://intranet.epa.gov/ocfo/policies/itcostacctg.htm. The section includes, as promised in my November 19, 2001 response to your preliminary finding outlines and position papers, several reports on FY 2002 spending for IT. OCFO plans to add instructional materials for system owners, funds control officers, and others to this page. To build on this year's experience, OCFO staff are working closely with OEI, the contracts community, headquarters SIRMOs, regional IRM branch chiefs, a regional comptroller, and others. For example, most regions are voluntarily piloting a method that uses two characters to classify their IT investment in greater detail than required by the PA. Results of the pilot are now being evaluated, and proposals are on the table to require a similar level of detail agency wide. Our goal is high quality cost accounting without overly burdensome and time consuming requirements. Sue Arnold 202-564-5192 can answer any questions. Attachment cc: Linda Combs Mike Ryan Mark Day Terry Ouverson Tim Rothwell John Gherardini Larry Wyborski Krista Mainess 68 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act OCFO RESPONSES TO OIG DRAFT RECOMMENDATIONS Chapter 6 of the Inspector General's Draft Report on Management of Information Technology Resources offers four recommendations for the CFO. OCFO's responses are below. Recommendation 6-1 - Develop consistent definitions of systems life cycle stages and IT costs to be used for contracting, accounting, IT systems, project management, and the capital planning investment control process. (Joint recommendation for the CIO, CFO and Assistant Administrator, OARM) Response - Comptroller Policy Announcement No. 01-10, "New Information Technology Accounting Requirements" (PA) includes these detailed definitions. To help ensure consistency across the Agency, OCFO has been an active participant in OEI's workgroup to update IRM Policy Manual 2100, Chapter 17 - System Life Cycle Management, since the workgroup's inception in November 2001. Recommendation 6-2 - Amend all current Agency software development contracts, and require that all future IT software development contracts be written to require a contractor to break out and separately report all IT software development costs by the system development life cycle. (Joint recommendation for the CIO, CFO and Assistant Administrator, OARM) Response - Attachment B of the PA requires that procurement documents show the life cycle phase, allowing software development costs to be easily rolled up for capitalization. Attachment A requires that project officers (POs), delivery order project officers (DOPO), and contracting officer technical representatives (COTRs) ensure proper IT coding on funding documents, proper allocation of IT activities on invoice payments, and proper classification of projects and systems under their control. Recommendation 6-3 - Develop consistent systems life cycle and IT costs definitions for revising EPA Directive 2100, and the interim IT activities policy guidance. (Joint recommendation for the CIO and CFO) Response - Please see response to Recommendation 6-1. 69 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Recommendation 6-4 - Complete a needs and feasibility assessment of alternatives to determine what types of project cost information and supporting documentation are needed for the capital planning investment control process and managing IT projects. Response - As stated above, OCFO is now implementing a structured plan to evaluate the cost information now required by the PA and to make appropriate refinements. We are working closely with OEI in the light of OMB's new CPIC requirements, as well as with OARM, headquarters SIRMOs, Regional IRM Branch Chiefs, representatives from the funds control and finance communities, and others. 70 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Appendix 4 Office of Air Quality Planning and Standards' Response to Draft Audit Report UNITED STATES ENVIRONMENTAL PROTECTION AGENCY Office of Air Quality Planning and Standards Research Triangle Park, NC 27711 May 28 2002 MEMORANDUM SUBJECT: Response to April 26, 2002 request for comments on Clinger-Cohen Act FROM: William T. Harnett, Director Information Transfer and Program Integration Division (MC-C304-03) TO: Patricia H. Hill, Director for Business Systems Office of the Inspector General for Audit (MC-2421) This memorandum responds to your April 26 request for comments on the IG's recently released draft report "EPA Management of Information Technology Resources under the Clinger-Cohen Act". The report primarily discusses how Office of Environmental Information (OEI) and the Chief Information Officer have implemented this important legislation. The report also refers to certain Agency data systems, such as Air Quality System (AQS). In this respect, the report mentions AQS in two places. One, on page 32, is in relation to an IG recommendation that a System Modernization Plan (SMP) be prepared for AQS and approved by the Assistant Administrator/Office of Air and Radiation (OAR). We generally agree with this and plan to revise the SMP and submit it for concurrence. The other reference is on page 9. In this case, we are uncertain of the scope of the issue and have copied the full paragraph from the draft to illustrate our uncertainty. "The slowly evolving and decentralized approach being used to develop an IT investment control structure has not been successful. EPA's approach allowed IT projects to be funded without proper justification, and in the absence of adequate management controls. EPA invested resources on outdated systems that did not maximize the efficiency or resolve long-standing problems, such as integration of environmental data. For example, the Air Quality System was funded $2.5 million for fiscal 2001, although planned modifications did not include adapting the system to function in conjunction with EPA's Central Data Exchange portal 71 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act 2 From the last sentence, it appears there is a concern that AQS was not a part of Central Data Exchange (CDX) in fiscal 01. However, given the preceding sentences, it appears there is also a concern that AQS is a project funded without proper justification and without management controls. In addition, it could be interpreted there is a concern that AQS is an outdated system. We do not believe the report provides an accurate characterization of AQS if all of these concerns are intended for AQS. With respect to the comment about AQS and the CDX, the AQS Information Technology (IT) budget proposal submitted in FY-01 did include our intent to work with OEI on a joint CDX pilot project in FY-02. In fact, OEI/OAR staff were actively meeting in FY-01 to develop a work plan which was submitted to the Quality and Information Council in late 2001 and approved in early 2002 (along with funding from the Agency's System Modernization Fund). Work is now underway. We also disagree with the IG comment that seems to imply that AQS is an outdated systems that does not maximize the efficiency or resolve long standing problems such as integration of environmental data. The AQS is an Oracle relational data base which is the Agency's recommended architecture for such applications. One benefit of Oracle systems is their ability to be integrated with data from other Oracle data bases (such as those being developed throughout the Agency). This technology is consistent with the Agency's approach for data integration; it is not outdated technology. If the report is intended to also portray AQS as a system with a lack of proper justification and absence of adequate management controls, material support for this conclusion is lacking in the narrative. We are hopeful the first two sentences of the above citation were not intended to apply to AQS. If they do apply, further explanation is essential. In either case, some editing of the paragraph is recommended. In summary, we believe this paragraph mischaracterizes the AQS system in many respects. I believe a conference call with you or your staff would be helpful. Again, thank you for the opportunity to comment and I look forward to discussing the matter with you at your earliest convenience. cc: J. Seitz, OAQPS T. Curran, OAQPS B. Kellam, ITPID E. Lillis, ITPID J. Summers, ITPID I. Spons R. Slade 72 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Appendix 5 Office of Solid Waste and Emergency Response's Response to Draft Audit Report UNITED STATES ENVIRONMENTAL PROTECTION AGENCY WASHINGTON, D.C. 20460 JUN 1 4 2002 OFFICE SOLID WASTE AND EMERGENCY RESPONSE MEMORANDTJM SUBJECT: OIG Draft Report "EPA's Management of Information Technology Resources Under the Clinger-Cohen Act" Audit Number 2001-0591 FROM: Marianne LamontHorinko Assistant Administrator TO: Kimberly Nelson Chief Information Officer (2823) The Office of Solid Waste (OSW) agrees in principle with the general spirit of the OIG report and concurs with the suggested future approaches to system development and project management in EPA. However, contrary to its portrayal in the draft report, we believe that RCRAInfo serves as a model for modular system development, rather than an example for how not to develop systems in our agency. The modular approach has enabled RCRAInfo to remain flexible to the changing needs of our constituent groups and allowed us to avoid some administrative pitfalls other projects have encountered. It has also eased the administrative burden. The modular approach uses the Program Area Analysis in its development of requirements for RCRAInfo, which is then approved by senior management before actual development occurs. This inevitably leads to RCRAInfo being in more than one stage of the system development life cycle. We made this choice intentional to allow the system to adapt in a timely, flexible manner to changing program requirements. Before the beginning of each major project within RCRAInfo, senior managers agreed on the need, and benefit, of continuing with that specific project. Senior managers also agreed on levels of funding for each project. On page 31, the report states that, "Despite several years of effort, management was still defining RCRAInfo requirementsWhile some requirements are still being defined for a few RCRAInfo modules, the majority of the RCRAInfo modules are well past this stage and in the development stage. OSW believes that the use of the Information Engineering model, combined with the separation of RCRAInfo into distinct modules that can be independently analyzed and developed, is an appropriate methodology to use for a large, complicated, and dynamic system such as RCRAInfo. Additionally, the report implies that work on the most crucial modules was put off while earlier modules were developed. The report fails to mention that EPA and its State partners deliberately approached each RCRAInfo module in a consensus order established by senior decision makers. To make the broad RCRA 73 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act analysis more useluL hlJA and the States decided which 3 areas were appropriate lor detailed analysis and on the order in which modules would be pursued. Staging the analysis in this manner allowed a number of improvements to move forward (e.g., one recommendation from an early module led to consolidating site information across three different mechanisms into a single form) while appropriate expertise (e.g., compliance personnel) could be directed at the last two modules on a separate track. The schedule also reflected the availability of key/personnel to work on modules. Finally, the following comments are offered regarding recommendations made specifically for RCRAInfo: Recommendation 5-5: Revise the Project Management Plan for the RCRAInfo project to make it equivalent to a System Management Plan (SMP) and update the document for planned system design changes and enhancements. In addition, the revised System Management Plan (SMP) should be formally approved by the Assistant Administrator for Solid Waste and Emergency Response to authorize funding for the IT investment and to ensure a system of accountability. Recommendation 5-7: Link the SMP to the Agency Clinger-Cohen Act submission documents and the Enterprise Architecture and planning documents. Recommendation 5-8: Manage project development efforts in accordance with the SMP, as updated, throughout the life cycle of the system, and retain the SMP for reference and review by the CIO or the CIO's designated review official. We feel the current development and management structure in place for RCRAInfo already meets the recommended actions and that no change is needed in that structure. RCRAInfo has a System Management Plan (SMP) in place, as well as a change and enhancement plan. In addition to the Capital Planning and Investment Control Proposal (CPIC) process, RCRAInfo adheres to a formal approval process for the Assistant Administrator for the Office of Solid Waste and Emergency Response to authorize funding for the IT investment and to ensure a system of accountability. cc: JeffWorthington William Ocampo Brion Cook Linda Travers Linda Garrison 74 Report No. 2002-P-00017 ------- EPA Management of Information Technology Resources Under The Clinger-Cohen Act Appendix 6 Report Distribution Headquarters Administrator Deputy Administrator Chief Financial Officer Assistant Administrator for Air and Radiation Assistant Administrator for Enforcement and Compliance Assurance Assistant Administrator for Environmental Information Assistant Administrator for Solid Waste and Emergency Response Associate Administrator for Congressional and Intergovernmental Relations Associate Administrator for Regional Operations and State/Local Relations Associate Administrator for Congressional and Legislative Affairs Associate Administrator for Communications, Education, and Public Affairs Agency Followup Official (2710) Agency Followup Coordinator (2724) Headquarters Library Office of Inspector General Inspector General Regional Offices Regional Administrators Regional Libraries Other General Accounting Office National Academy of Public Administration 75 Report No. 2002-P-00017 ------- |