§ < 30 V ^tosrx *1 PRQ^ o 2 ul o OFFICE OF INSPECTOR GENERAL Catalyst for Improving the Environment Audit Report Improved Management Practices Needed to Increase Use of Exchange Network Report No. 2007-P-00030 August 20, 2007 New Jersey Department of Environmental Protection Tennesson Dcpjrtrrwnt of Environment and Conservation INTERNET EPA National Computer Center Research Triangle Park, NC New Mexico Environment Department New Mexico Mobile Air Monitoring Trailer North Carolina Department of Environment and Natural Resources South Carolina Department of Health & Environmental Control Kentucky Department for Environmental Protection ------- Report Contributors: Rudolph M. Brevard Cheryl Reid Jefferson Gilkeson Scott Sammons Abbreviations AFS Air Facility System AQS Air Quality System CDX Central Data Exchange ENLC Exchange Network Leadership Council EPA U.S. Environmental Protection Agency IT Information Technology NOB Network Operations Board OIG Office of Inspector General OMB Office of Management and Budget PCS Permit Compliance System RCRA Resource Conservation and Recovery Act RCRAInfo Resource Conservation and Recovery Act Information System SDWIS Safe Drinking Water Information System Cover photos: Clockwise from top: Conceptual representation of the Exchange Network using photos from: • Kentucky Department for Environmental Protection • New Jersey Department of Environmental Protection • EPA National Computer Center, Research Triangle Park, North Carolina • North Carolina Department of Environment and Natural Resources • South Carolina Department of Health and Environmental Control • New Mexico Environment Department's Mobile Air Monitoring Trailer • New Mexico Environment Department • Tennessee Department of Environment and Conservation (Photos courtesy of EPA and the States featured) ------- ^tD S7-/5>- / A \ EnvJ™nme"tal Protection Agency 2007-P-00030 I JUL | Office of Inspector General August 20,2007 " o ^ ¦ .0 • u¦ o¦ ^11 vii vJi 111 ic 11 id 1 n uiwvii bjgjui \ Office of Inspector General \S|72 1 At a Glance Why We Did This Review We sought to determine: • What barriers exist that prevent the National Environmental Information Exchange Network (Network) from achieving maximum usage, and steps the U.S. Environmental Protection Agency (EPA) can take to overcome them. • Whether EPA has developed Network performance measures that align with its Strategic Plan. • How EPA could improve key system development processes for analyzing costs and ensuring Network use for new systems and upgrades. • How EPA could assist the Network governance bodies in accomplishing their missions. Background The Exchange Network is EPA's approach (and expected preferred method) for the exchange of environmental data among Network partners. As of January 2007, 48 States and 2 tribes used the Network. EPA has invested more than $162 million on the Network. For further information, contact our Office of Congressional and Public Liaison at (202) 566-2391. To view the full report, click on the following link: www.epa.aov/oia/reports/2007/ 20070820-2007-P-00030.pdf Catalyst for Improving the Environment Improved Management Practices Needed to Increase Use of Exchange Network What We Found Although EPA established a partnership with the Exchange Network's governance bodies to assist them with accomplishing Network initiatives, more improvements are needed to ensure Network partners folly utilize the Network. These partners include EPA, States, tribes, territories, and other parties with whom EPA and States exchange information. EPA should improve its methods for selecting and prioritizing which data flows to implement. EPA also needs to take further steps to complete measurements of Network initiatives to ensure investments are delivering expected results. In addition, EPA needs to improve its internal system development practices to ensure EPA offices perform cost benefit analyses for new or upgraded environmental systems. Further, EPA should strengthen its policies to define when offices should utilize the Network for receiving environmental information. The Exchange Network Business Plan stresses the importance of having an effective collaborative partnership between EPA, the Network governance bodies, and the Network partners. Since EPA intends for the Exchange Network to become the preferred method for exchanging environmental data and foresees expanding the Network, EPA should take steps to improve Network use. Without taking action, EPA would not know when or whether its partners would adopt the Network as the preferred method to share data with EPA. As such, EPA investments in the Network would not yield the desired outcomes. What We Recommend We made various recommendations, including that the EPA Office of Environmental Information: • Execute the Exchange Network Marketing and Communications plan and evaluate data flows for Network implementation, • Develop a new plan for completing the Exchange Network performance measures project, • Develop policies and procedures to guide program offices to use the Network and conduct Exchange Network Cost Benefit Analysis, and • Include the Exchange Network in the Enterprise Architecture. In general, the Agency agreed with the report's findings and recommendations. We have summarized the Agency's comments in the following report chapters and included the Agency's complete response in Appendix C. ------- ^tDsrx | \ UNITED STATES ENVIRONMENTAL PROTECTION AGENCY % I WASHINGTON, D.C. 20460 PRO^ OFFICE OF INSPECTOR GENERAL MEMORANDUM SUBJECT: FROM: TO: August 20, 2007 Improved Management Practices Needed to Increase Use of Exchange Network Report No. 2007-P-00030 for Patricia H. Hill Assistant Inspector General for Mission Systems Linda A. Travers Deputy Assistant Administrator for Environmental Information This is our report on the subject audit conducted by the Office of Inspector General (OIG) of the U.S. Environmental Protection Agency (EPA). It contains findings that describe the problems the OIG has identified and corrective actions the OIG recommends. This report represents the opinion of the OIG and does not necessarily represent the final EPA position. Final determinations on matters in this report will be made by EPA managers in accordance with established audit resolution procedures. The estimated cost of this report - calculated by multiplying the project's staff days by the applicable daily full cost billing rates in effect at the time - is $665,051. Action Required In accordance with EPA Manual 2750, you are required to provide a written response to this report within 90 calendar days. You should include a corrective actions plan for agreed upon actions, including milestone dates. We have no objections to the further release of this report to the public. This report will be available at http://www.epa.gov/oig. If you or your staff have any questions, please contact Rudolph Brevard, Director for Information Resources Management Assessments, at (202) 566-0893 or brevard.rudolph@epa. gov. ------- Improved Management Practices Needed to Increase Use of Exchange Network Table of C Chapters 1 Introduction 1 Purpose 1 Background 1 Noteworthy Achievements 3 Scope and Methodology 3 2 EPA Needs to Implement Exchange Network Business Plan Elements 5 EPA Needs to Market the Network's Business Value to the States 5 EPA Needs to Modify the Network's Change Management Process 6 Recommendations 7 Agency Comments and OIG Evaluation 7 3 EPA Needs to Use a Formal Process to Select and Prioritize Data Flows 8 EPA Needs to Invest in the Most Beneficial Network Initiatives 8 Recommendation 10 Agency Comments and OIG Evaluation 10 4 EPA Needs to Measure Progress Meeting Exchange Network Performance Objectives 11 Performance Measures Workgroup Needs More Structure 11 Recommendations 12 Agency Comments and OIG Evaluation 12 5 EPA Needs to Improve Exchange Network System Development Processes 13 Program Offices Need to Perform Cost Benefit Analysis 13 EPA Needs a Policy Defining When to Use the Network 14 Recommendations 15 Agency Comments and OIG Evaluation 15 Status of Recommendations and Potential Monetary Benefits 16 - continued - ------- Improved Management Practices Needed to Increase Use of Exchange Network Appendices A Detailed Scope and Methodology 17 B Federal and Agency Criteria 21 C Agency Response to Draft Report 23 D Distribution 28 ------- Chapter 1 Introduction Purpose We evaluated the U.S. Environmental Protection Agency's (EPA's) implementation of electronic data collection initiatives. We did this by reviewing a sample of EPA data collection projects (called data flows) related to the National Environmental Information Exchange Network (Exchange Network, or Network). Specifically, we sought to determine: • What internal EPA or external stakeholder barriers exist that prevent selected Exchange Network projects from achieving maximum usage by Network partners and steps EPA can take to overcome identified barriers. • Whether EPA has developed performance measures that align with Exchange Network objectives stated in the Agency's Strategic Plan. We also assessed how EPA could improve processes for: • Performing cost benefit analysis of Exchange Network projects, and establishing policies and procedures to ensure program offices design systems, when applicable, to use the Exchange Network when developing new or upgrading existing systems. • Assisting the two Exchange Network governance bodies - the Exchange Network Leadership Council (ENLC) and its operational sub-group, the Network Operations Board (NOB) - in accomplishing their missions. Background The Exchange Network is a secure Internet- and standards-based approach for exchanging environmental data and improving environmental decisions. EPA, State environmental departments, U.S. tribes, and territories have acted as partners in building the Network. By using interconnected computers (called Network nodes), the Network provides access to, and exchange of, environmental data. As shown in Figure 1-1, partners use their Network nodes and standardized data templates to exchange data over the Internet with other Network partners. This node-to-node exchange of data is intended to increase efficiency through automation and reduce reporting burden. Over time, EPA expects the Network to become the preferred method used by Network partners to exchange environmental data with EPA and other Network partners. 1 ------- Figure 1-1: Exchange Network Overview Partner A e.q. 51a1e hternet Partner B Partner D (e.g. EPAj Network Node Data Exchange Template Source: www.exchanaenetwork.net The governance of the Network is a partnership between EPA and the various Network partners. Each of the governance bodies - ENLC and NOB - include EPA, State, and tribal representatives. The ENLC provides leadership to the Exchange Network and is responsible for the Network's overall policies and strategic direction. The NOB oversees day-to-day operation of the Network's shared infrastructure and identifies, prioritizes, and resolves Network operational issues. The ENLC and NOB have the authority to establish workgroups as needed to organize and execute their responsibilities. The Environmental Council of the States plays a strong coordinating role for participating State executives. EPA's Office of Environmental Information manages the Central Data Exchange (CDX). This exchange represents EPA's node on the Exchange Network and is the portal used by Network partners to share environmental data with EPA. EPA program offices are responsible for ensuring, when applicable, that their information systems are able to accept data sent by Network partners via the Network. This may require Program Offices to incorporate additional technologies in order to receive data sent by Network Partners over the Exchange Network. Tables 1-1 and 1-2 provide key Exchange Network statistics as of January 2007. Table 1-1: Exchange Network Funding Data $163.4 million The total amount of funding EPA provided internally and to States, tribes, and territories for developing the Exchange Network. $101.9 million The amount of grants awarded to States, tribes, and territories to fund Exchange Network initiatives. $61.5 million The amount of funding EPA provided internally to develop and maintain the CDX. Source: OIG compilation and analysis of information from EPA and www.exchanaenetwork.net 2 ------- Table 1-2: Exchange Network Usage Data 15 The number of Exchange Network data flows EPA has available for Network partner use; includes 7 regulatory data flows. 48 States The number of Exchange Network partners using the Network to share 2 Tribes environmental information with EPA and other Network partners. 37 States The number of States using the Network to share regulatory data with EPA. Source: OIG compilation and analysis of information from www.exchanqenetwork.net Noteworthy Achievements All six State environmental agencies we visited cited EPA grant funding as the primary reason enabling their use of the Exchange Network. States also cited the promotion of data standards and enablement of State-to-State exchanges as actual or expected benefits from Network participation. In addition to the seven available regulatory data flows, EPA implemented non- regulatory data flows (e.g., Water Quality Exchange) and Agency outbound data flows to States (e.g., Toxic Release Inventory). EPA representatives indicated the Toxic Release Inventory data flow would help (1) reduce States' data collection time and effort, (2) improve data quality, and (3) ensure the consistency of EPA and State database records. Scope and Methodology We conducted this audit in accordance with Government Auditing Standards, issued by the Comptroller General of the United States. We conducted this audit from January through December 2006 at EPA Headquarters in Washington, DC; Research Triangle Park, North Carolina; and six State environmental agencies. We reviewed Exchange Network background information, as well as relevant Federal and Agency policies and procedures related to managing Information Technology (IT) investments. We interviewed EPA and State employees responsible for overseeing and implementing Exchange Network projects. We also interviewed State employees who work in or manage State air, water, or land programs. Appendix A includes information on management controls, report limitations, and our selections of Exchange Network projects and States visited. Appendix A also provides the specific scope and methodology applied for each audit objective. Appendix B contains a list of relevant Federal and Agency policies and procedures as well as other criteria documents we reviewed. There were no pertinent issues that required followup from prior audit reports. 3 ------- We conducted a limited review of the two Exchange Network governance bodies in five key strategic areas. We analyzed the results of a questionnaire sent to each body, conducted interviews with representatives from each body, and reviewed each of the body's charter. Our analysis did not discover areas where EPA could further assist the governance bodies in the areas under review. Therefore, we did not pursue this area during field work. 4 ------- Chapter 2 EPA Needs to Implement Exchange Network Business Plan Elements EPA should take additional steps, as per its Exchange Network Business Plan, to assist Network partners in using the Network. We found that: • Partners did not fully understand the benefits of using the Network. • Partners viewed the Network as an IT project instead of a new way to manage environmental information. • Partners incurred added costs and lost time because EPA did not always communicate Network changes or fully test Network implementation tools. This occurred because EPA had not implemented a plan to communicate the business value of using the Network. EPA also had not put in place processes to communicate changes in data requirements or fully test Network implementation tools. The Exchange Network Business Plan stresses the importance of communication with Network stakeholders. These key communication processes help Network partners initiate, expand, and sustain Network usage. Due to the issues noted, EPA does not know when or whether its partners would adopt the Network as the preferred method to share regulatory data with EPA. EPA Needs to Market the Network's Business Value to the States EPA officials indicated they provided technology transfer assistance for the Network through its technology team and contractors. This included conferences, and face-to-face seminars, and one-to-one assistance to promote and explain Exchange Network technology to State IT staff and their contractors. EPA also promoted the benefits of the Network at national meetings with State environmental commissioners and at EPA program office conferences. However, State air, water, and land program managers we interviewed did not fully understand the Network, its technologies, or anticipated benefits. During our visits to six State environmental offices, we interviewed State employees who work in or manage their State's air, water, or land programs. We also interviewed State IT employees responsible for implementing Network data flows. We asked both groups to provide examples of benefits from Network participation. 5 ------- The IT employees described benefits such as improved data quality and increased automation. However, State program office employees often could not cite specific programmatic benefits provided by the Network. They also stated that they did not know how the Network could be used in their program, nor what programmatic benefits it could provide. Further, employees from all six State environmental agencies said EPA needed to do a better job marketing the business value of Network participation. The Exchange Network Business Plan addresses the need to market ways that the Network adds value, and links partner activities with Network goals and objectives. Often the impetus to implement a Network data flow occurred when a State IT employee initiated an Exchange Network grant. Therefore, the decision to participate in the Network was driven by the IT department instead of the State program office. Because of this and also because EPA promoted technical benefits, such as improved data quality and data availability, two State program office employees and the ENLC indicated that the Network is viewed as an IT project rather than an information management initiative. As a result, State program office employees have not fully utilized the Network. It is also unknown whether or when States will adopt the Network as the preferred method for inter-governmental transfer of environmental data. Subsequent to audit field work, in December 2006 EPA published the Exchange Network Communications and Marketing Plan. This document addresses the need to communicate the specific benefits of Network participation, in non-technical terms, to State environmental program officials. This document also provides the context, framework, and guidance for the implementation of all Exchange Network communications, initiatives, and activities. The ENLC recently acknowledged that the Network is seen by senior managers as a technology project instead of as an information management initiative. An ENLC official indicated that the ENLC is conducting outreach to States to understand their business needs and to foster relevant growth and application of the Network. EPA Needs to Modify the Network's Change Management Process Representatives from two State agencies said they incurred added costs and lost time because EPA did not always communicate Network changes or fully test Network implementation tools. Employees from all six State agencies visited cited instances when they had attempted, but were unable, to use the Network to send data to EPA. They indicated this occurred because EPA did not communicate changes made to data requirements or error checking routines. State agency employees also cited instances when Network tools (developed to accelerate data flow implementation) did not work. These conditions impeded States' efforts to sustain and expand Network usage. 6 ------- These conditions occurred because EPA did not implement, in a timely manner, a change management process for the Network. In addition, EPA lacked a formal process to communicate changes to Network partners, or to fully test implementation tools. The Exchange Network Business Plan identifies the need to develop a formal change management process for communicating and controlling changes that can adversely affect partners' ability to use the Network. This change management process is important to the successful and sustained usage of the Exchange Network. Because these processes were not in place, States incurred rework costs and experienced delays in using the Network to share data with EPA. Subsequent to audit field work, EPA adopted and published formal change management policies and procedures for the Exchange Network. These documents define the Network change management practices and include processes for communicating changes that affect partner implementation efforts. However, these policies and procedures do not include detailed step- by-step processes for testing and certifying Exchange Network implementation tools. Recommendations We recommend that the Deputy Assistant Administrator for Environmental Information have the Director, Office of Information Collection: 2-1 Execute recently developed Exchange Network Communications and Marketing plan elements that include actively promoting the business value of participating in Network initiatives to EPA and partner environmental program managers. 2-2 Modify Exchange Network change management policies and procedures to include step-by-step processes for fully testing and certifying all implementation tools before release to the Exchange Network community. Agency Comments and OIG Evaluation The Agency agreed with the findings and recommendations. Management indicated it would ensure it provides recommendations to promote the business value of participating in Network initiatives to the ENLC and indicated that the Network Technical Group, a subgroup of the Network Operations Board, would develop procedures for testing and certifying all implementation tools. 7 ------- Chapter 3 EPA Needs to Use a Formal Process to Select and Prioritize Data Flows EPA did not always implement data flows most beneficial to Network partners. The Network Business Plan provides criteria for evaluating Network data flows. These criteria favor using flows with stable data requirements, and those most likely to be implemented by Network partners. However, EPA did not follow these criteria and instead decided to first implement all regulatory data flows. EPA officials said this approach would result in the greatest reduction in States' reporting burden to EPA. However, States already had a way to provide regulatory data to EPA. Therefore, some regulatory data flows were implemented that have little State usage. Using a formal process to guide Exchange Network investment decisions increases the likelihood that public funds are effectively spent. Soliciting Network partner input increases Network usage by investing in those initiatives most needed by Network partners. By not doing this, EPA may direct public funds to Network initiatives that may not be most beneficial to partners or that do not maximize Network usage. EPA Needs to Invest in the Most Beneficial Network Initiatives EPA did not timely solicit State program offices' input or follow Network Business Plan guidance to select and prioritize investments in Network data flows. State employees cited, as an area for improvement, that EPA should solicit their input to understand how the Network can address their needs. We found that five of the six States we visited wanted to implement a data flow but it was not available. For example, two States we visited had a keen interest in electronically providing water permit data to the Permit Compliance System (PCS), one of the systems in our sample. However, a PCS network data flow will not be available until 2008. Also, three of the available Network data flows in our sample had little State participation. For example, for the Resource Conservation and Recovery Act Information System (RCRAInfo) data flow, only four States regularly use the Exchange Network to send Resource Conservation and Recovery Act (RCRA) data to EPA. As reasons, State employees cited a lack of identifiable benefits and the difficulty implementing this data flow. Further, the one State visited that uses the Network to provide RCRA data to EPA reported no benefits. In addition, EPA RCRA program officials had not documented any benefits to substantiate modernizing the RCRAInfo system to accept data through the Network. 8 ------- EPA did not follow Network Business Plan guidance to select and prioritize investments in Network data flows. The Network Business Plan, endorsed by EPA senior management officials, contains specific criteria for determining which data flows are most suitable for Network use. These criteria include determining whether the Network will provide specific benefits by: • Making the data exchange more efficient by reducing or eliminating manual intervention, such as scheduling, resubmissions or security. • Providing higher data quality due to more efficient error checking and/or earlier detection of errors/discrepancies. The criteria also state that certain data flows are more appropriate for the Network than others. Important considerations include the extent to which data flows are likely to capture the Network effect by engaging multiple partners, as well as the volatility of data and frequency of exchange. Agency officials and representatives said implementing regulatory flows would produce the greatest reduction in States' reporting burden. Accordingly, EPA invested resources to implement these data flows on the Exchange Network. Additionally, the Network Business Plan provides, for planning purposes, a scenario in which, by the year 2010, a majority of States will use the Network to meet most of their regulatory reporting requirements. As shown in Figure 3-1, 37 of 50 States use the Network to share regulatory data with EPA. However, only 9 of 50 States implemented 3 or more regulatory data flows and no State uses all 7 of the available regulatory data flows. As such, only a small number of States use the majority of available regulatory data flows. The lower-than-projected usage of regulatory data flows partly stems from States already having an acceptable way to meet their regulatory reporting requirements. EPA had not always considered this when deciding to implement the regulatory data flows. As such, EPA implemented regulatory data flows although the data flows did not meet the investment selection criteria. By not using their own approved process to select and prioritize investments in Network initiatives, EPA directed public resources to some Network initiatives that may have produced few, if any, benefits. Given that EPA continues to expand the use of the Network beyond receiving regulatory Figure 3-1: State Participation in Regulatory Data Flows ~ three or more ¦ one to two ~ none Source: OIG analysis of Exchange Network usage data from www.exchanqenetwork.net 9 ------- data, EPA should apply the Network Business Plan criteria when deciding which future Network projects to fund. This should result in the better use of public funds and increased partner usage of the Network. Recommendation We recommend that the Deputy Assistant Administrator for Environmental Information: 3-1 Work with Exchange Network governance bodies to develop and implement a process that uses Network Business Plan criteria to evaluate data flows for future Network implementation. Agency Comments and OIG Evaluation The Agency agreed with the finding and recommendation. Management indicated it would develop a process jointly with the Exchange Network's governance bodies. 10 ------- Chapter 4 EPA Needs to Measure Progress Meeting Exchange Network Performance Objectives Although EPA defined performance measures related to achieving Exchange Network objectives in EPA's strategic plan, it has not measured progress made. These objectives define how the Network would provide wider information sharing and contribute to better decision-making. These objectives also define how the Network reduces the burden on those that provide and access information. EPA shares responsibilities for developing and reporting the results of Network performance measures with a workgroup comprised of EPA and Network partners. However, this workgroup lacks a charter and defined roles and duties for members. This lack of structure led to the Agency missing milestones in its plan for the Network's performance measures project. Without collecting performance data on key tasks and milestones, EPA does not know whether Network investments achieve their stated objectives or are implemented within cost and schedule estimates. Performance Measures Workgroup Needs More Structure Although EPA developed a comprehensive work plan for the Exchange Network performance measures project, key tasks remain to be completed and milestone dates have been missed. The performance measures workgroup has established metrics and Agency representatives stated they developed measurement survey instruments to measure progress. However, as of January 31, 2007, the workgroup had not collected data to measure progress made in attaining key Network objectives. The Work Area Task Plan: Performance Measurements - Exchange Network outlines development of outcome-oriented performance measures for the Exchange Network. This plan outlines the collection, analysis, and interpretation of baseline indicator data to measure performance over time. These measures, along with EPA's performance measurement efforts, focus on showing progress made toward meeting the goals and objectives in EPA's 2003-2008 strategic plan. EPA representatives indicated they initially established the performance measures workgroup to complete the work outlined in the task plan. EPA representatives indicated they later expanded the workgroup to include Network partners because EPA believed it needed external input to measure certain parts of the plan. EPA now shares the oversight of the workgroup with the Network governance bodies. However, neither EPA nor the Network 11 ------- governing bodies have processes in place for setting up and overseeing ad-hoc workgroups working on Network projects. This led to establishing the performance measures workgroup without a charter and defined roles and responsibilities. Without performance measures, management is seriously hindered in its efforts to improve program efficiency and effectiveness. The Agency cannot ensure (1) the Exchange Network investment supports the Agency's Strategic Plan, (2) the investments progress in accordance with planned performance baselines, and (3) projects are modified and adjusted accordingly. Subsequent to the draft report, EPA indicated it took action to correct this finding. It disbanded the metrics workgroup and moved the responsibilities under the Network Partnership and Resources Group. We re-evaluated the group's charter and agree with the Agency that it includes Exchange Network performance measures oversight responsibilities. Recommendations We recommend that the Deputy Assistant Administrator for Environmental Information have the Director, Office of Information Collection: 4-1 Develop a new milestone plan for completing the Exchange Network performance measures project. We also recommend that the Deputy Assistant Administrator for Environmental Information work with the Exchange Network governance bodies to: 4-2 Develop procedures for establishing ad-hoc workgroups for Exchange Network projects. Agency Comments and OIG Evaluation The Agency agreed with the findings and recommendations and provided the OIG with corrective actions to address the problems related to the Exchange Network performance measure responsibilities. The Agency indicated the performance measures project is now on a schedule under the control of the Network Partnership and Resources Group, and indicated it will develop procedures for new ad-hoc workgroups. We modified the report and our final recommendations accordingly. 12 ------- Chapter 5 EPA Needs to Improve Exchange Network System Development Processes EPA should strengthen system development processes related to Exchange Network initiatives. EPA's Capital Planning and Investment Control policy and the Exchange Network Business Plan provide the framework for planning, putting in place, or continuing an e-business Information Technology investment. However, we found that: • EPA program offices in our sample had not performed a cost benefit analysis on Exchange Network technologies before investing in Network projects. • EPA had not published policies to guide offices to use the Network when developing or upgrading systems. These issues exist because EPA's cost benefit analysis process does not contain steps to ensure full consideration of Network technologies. In addition, EPA's Enterprise Architecture does not formally recognize the Exchange Network. As such, these weaknesses can lead to investment costs outweighing benefits or investments not aligning with EPA's IT plans. Program Offices Need to Perform Cost Benefit Analysis EPA invested in Exchange Network technologies without determining if benefits realized would exceed costs incurred. EPA's Capital Planning and Investment Control Procedures for the Office of Management and Budget (OMB) Exhibit 300 (CPIC procedures) requires the development of a cost benefit analysis prior to initiating, continuing, or implementing an IT investment. The procedure states that the analysis must contain three technical alternatives, with one alternative being "as is." It also states that each viable technical approach should be included as an alternative. A cost benefit analysis is required, regardless of cost, if the IT investment uses e-business technologies. Program offices did not conduct a cost benefit analysis to consider Network technologies because Agency CPIC procedures lack specific guidance on evaluating data transfer protocols and Web services. Program office representatives said they included Exchange Network technologies in modernization efforts because the Agency was moving in that direction. Therefore, the offices did not evaluate whether other system development alternatives were more cost effective. Program office representatives also said they did not include Network analysis because it was a small part of their 13 ------- overall modifications. The Office of Environmental Information indicated it is planning to develop a Return on Investment tool that will allow program offices to determine the value of using EPA Exchange Network core services. However, it has yet to take steps to develop the tool. A properly developed cost benefit analysis establishes a baseline used to measure future progress. This baseline is a key tool for estimating benefits and costs. The post implementation review needs this baseline to validate benefits and costs, as well as to document effective management practices. Without a properly developed cost benefit analysis, the review might not have the information necessary to determine if investments are performing as intended. This may lead to projects having costs that exceed benefits or not meeting the needs for which the projects were initiated. During field work, EPA produced a CDX return on investment report on six data flows, and indicated it shared the results with representatives in four program offices. However, EPA needs to complete its plans to develop a tool offices can use to evaluate their applications in regard to Network technologies. EPA Needs a Policy Defining When to Use the Network Although EPA intends for the Network to become the preferred method for exchanging environmental data, EPA does not have a policy to guide program offices to use Exchange Network technologies, when applicable, during system development and modernization. EPA has not finished developing guidance to guide program offices to implement Network technologies in their systems. Also, EPA does not formally recognize the Exchange Network in the Agency's Enterprise Architecture, which requires program offices to adhere to the current Agency IT plans. By completing these two critical processes, EPA could ensure maximum consideration and use of the Network. This would also reinforce procedures to guide program offices on the use of the Exchange Network. Not guiding program offices on the use of the Exchange Network may lead to realizing less than maximum business value of Network investments. This may also create redundancy resulting from disparate planning and development efforts. As indicated earlier, CDX is EPA's node on the Exchange Network and CDX is incorporated in the Agency's Enterprise Architecture. EPA officials stated they recognize the Exchange Network in the Enterprise Architecture through the use of CDX. However, some EPA program offices are using the CDX for data exchanges without using Exchange Network technologies. Therefore, if EPA wants to ensure full consideration of the Exchange Network as the preferred method for exchanging environmental data, EPA should formally recognize the Exchange Network in the Enterprise Architecture and complete guidance that helps ensure offices implement the Network technologies in 14 ------- their systems. This would help EPA ensure offices consider the Exchange Network and other various data sharing alternatives when replacing or upgrading their systems. Recommendations We recommend that the Deputy Assistant Administrator for Environmental Information have the Director, Office of Technology Operations and Planning: 5-1 Publish standards that specify when EPA program offices must use the Exchange Network when modernizing or developing applications. The standards should also specify the processes EPA offices must follow when the office cannot adhere to the established standards or select an alternate technological solution to the one prescribed. 5-2 Include the Exchange Network and related technologies as part of the Agency's Enterprise Architecture. We also recommend that the Deputy Assistant Administrator for Environmental Information have the Director, Office of Information Collection: 5-3 Complete its plans to develop a tool offices can use to evaluate their applications in regard to Network technologies. Agency Comments and OIG Evaluation In general, the Agency agreed with the findings and plans to take steps to address the recommendations. Management did not concur with our original recommendation 5-1 that recommended the Agency include Exchange Network Cost Benefit Analysis procedures in the Capital Planning and Investment Control process. Management did not believe that the Agency's Capital Planning and Investment Control process is the appropriate place to include details for Exchange Network technologies as part of the cost benefit analysis, because not all information systems will be using the Exchange Network. However, our review disclosed EPA offices did not evaluate the cost and benefits for using the Exchange Network, although these offices invested in the technology. As such, EPA should develop guidance to ensure that all relevant costs are identified and considered when investing in the Exchange Network. Subsequent to the Agency's response to the draft report, Agency officials acknowledged the need to establish standards within the Agency's policy framework that outline when the Exchange Network should be used, and concurred with our recommendation that the Agency take steps to establish them. Where appropriate, we modified the report and the final recommendations. 15 ------- Status of Recommendations and Potential Monetary Benefits RECOMMENDATIONS POTENTIAL MONETARY BENEFITS (In $000s) Rec. Page No. No. Subject Status1 Action Official Planned Completion Date Claimed Amount Agreed To Amount 2-1 7 Have the Director, Office of Information Collection, execute recently developed Exchange Network Communications and Marketing plan elements that include actively promoting the business value of participating in Network initiatives to EPA and partner environmental program managers. 2-2 7 Have the Director, Office of Information Collection, modify Exchange Network change management policies and procedures to include step-by-step processes for fully testing and certifying all implementation tools before release to the Exchange Network community. 3-1 10 Work with Exchange Network governance bodies to develop and implement a process that uses Network Business Plan criteria to evaluate data flows for future Network implementation. 4-1 12 Have the Director, Office of Information Collection, develop a new milestone plan for completing the Exchange Network performance measures project. 4-2 12 Work with the Exchange Network governance bodies to develop procedures for establishing ad-hoc workgroups for Exchange Network projects. 5-1 15 Have the Director, Office of Technology Operations and Planning, publish standards that specify when EPA program offices must use the Exchange Network when modernizing or developing applications. The standards should also specify the processes EPA offices must follow when the office cannot adhere to the established standards or select an alternate technological solution to the one prescribed. 5-2 15 Have the Director, Office of Technology Operations and Planning, include the Exchange Network and related technologies as part of the Agency's Enterprise Architecture. 5-3 15 Have the Director, Office of Information Collection, complete its plans to develop a tool offices can use to evaluate their applications in regard to Network technologies. Deputy Assistant Administrator for Environmental Information Deputy Assistant Administrator for Environmental Information Deputy Assistant Administrator for Environmental Information Deputy Assistant Administrator for Environmental Information Deputy Assistant Administrator for Environmental Information Deputy Assistant Administrator for Environmental Information Deputy Assistant Administrator for Environmental Information Deputy Assistant Administrator for Environmental Information 06/27/07 1 0 = recommendation is open with agreed-to corrective actions pending; C = recommendation is closed with all agreed-to actions completed; U = recommendation is undecided with resolution efforts in progress 16 ------- Appendix A Detailed Scope and Methodology Management Controls and Report Limitations We evaluated the following two management controls related to the effectiveness and efficiency of program operations: • EPA's controls over the effectiveness and efficiency of Electronic Initiatives and their interactions with the governing bodies. • The management control structure of the governing bodies to determine if procedures have been implemented to meet their objectives and included appropriate stakeholders. We did not review the provisions of Network contract or grant agreements and safeguarding of resources. Therefore, the user of this report would not be able to determine whether (1) funds awarded for Exchange Network initiatives were spent according to contract or grant provisions, or (2) EPA management took adequate steps to safeguard Agency- controlled resources. Selection of Exchange Network Data Flows We surveyed EPA program offices and regions to identify all Exchange Network projects. We selected only those Exchange Network projects that: (1) represent one of the nine regulatory Network data flows, and (2) use the Exchange Network to flow data from Network partners to EPA. This resulted in an initial sample of eight Exchange Network projects in four EPA program offices. We later reduced our sample size (due to time and resource constraints) to five Exchange Network projects within the four program offices. Table A-l provides a listing of selected Exchange Network data flows and EPA program offices, and indicates whether the data flow was available for implementation by Network partners. Table A-1: Selected Exchange Network Data Flows Exchange Network Data Flow Acronym Program Office Available for Implementation Air Quality System AQS Office of Air and Radiation X Air Facility System AFS Office of Enforcement and Compliance Assurance Permit Compliance System PCS Office of Enforcement and Compliance Assurance Resource Conservation and Recovery Act Information System RCRAInfo Office of Solid Waste and Emergency Response X Safe Drinking Water Information System SDWIS Office of Water X Source: OIG compilation of EPA data 17 ------- Selection of States for Site Visits We used information on the Exchange Network Web site (www.exchangenetwork.net) to select States for our site visits. We categorized States into two groups: (1) States that had not yet implemented an Exchange Network node, and (2) States that had done so. Next, we assigned those States with active Network nodes into sub-groups based on their level of participation (high, occasional, low, or no) in available data flows. We defined the various levels as such: High Exchange Network Participation: States that use the Network to provide data to EPA using two or more of the available regulatory data flows, at least one of which is from Table A-l. Occasional Exchange Network Participation: States that have used the Network to provide data to EPA using one or more of the available regulatory data flows, at least one of which is from Table A-l. However, these States no longer regularly use the Network and related data flow(s) to provide data to EPA. Low Exchange Network Participation: States that use the Network to share data with EPA using one or more of the available Network data flows; however, these States have not implemented, and are not in the process of implementing, any of the data flows from Table A-l . No Exchange Network Participation: States that received funds from the Exchange Network Grant Program but have not yet implemented a Network node. We selected three States from each of the above four categories. Using this initial sample of 12 States, we selected 6 States for site visits. These six States included at least one member from each of the four categories. Table A-2 identifies the six State environmental agencies we visited. Table A-2: State Environmental Agencies Selected State Agency Kentucky Department for Environmental Protection New Jersey Department of Environmental Protection New Mexico Environment Department North Carolina Department of Environment and Natural Resources South Carolina Department of Health and Environmental Control Tennessee Department of Environment and Conservation Source: EPA OIG 18 ------- Exchange Network Internal and External Barriers To identify internal (EPA) barriers to Exchange Network participation, we focused on determining why two of the data flows in our sample (AFS and PCS) were unavailable for partner implementation. We did this by interviewing EPA Office of Environmental Information and program office staff to determine: (1) current status and time frame for implementing these two data flows, and (2) additional steps EPA could take (if practical) to accelerate implementation of these data flows. To identify external (partner) barriers, we focused on determining conditions preventing States from implementing a Network node as well as conditions preventing States with active Network nodes from implementing the three operational data flows (AQS, RCRAInfo, and SDWIS) in our sample. We determined conditions preventing States from implementing a Network node by interviewing EPA Office of Environmental Information and State personnel and identifying factors preventing the State from implementing a Network node. Based on these interviews, we determined additional steps EPA could take to assist the State in implementing a Network node. We determined conditions preventing States with active nodes from participating in the AQS, RCRAInfo, and SDWIS data flows by interviewing EPA Office of Environmental Information, applicable EPA program office, and State personnel. Based on these interviews, we determined additional steps EPA could take to help States implement the AQS, RCRAInfo, and SDWIS data flows. Performance Measures To evaluate performance measures, we first reviewed relevant EPA and Federal criteria to determine what guidelines exist related to measuring performance of IT systems. We then reviewed EPA's strategic plan to determine the Agency's goals for the Exchange Network. Next, we interviewed representatives and officials from the EPA Office of Environmental Information to determine work performed related to Exchange Network performance measures and how this work aligned with EPA's goals for the Network. We reviewed documents related to the Exchange Network performance measures project to determine validity and completeness. Exchange Network Process Improvements To evaluate how EPA could improve key Exchange Network processes, we reviewed three areas: • Cost Benefit Analysis (to select and prioritize the implementation of Exchange Network projects) - We reviewed documents from the selected program offices for each respective system to determine whether: o a cost benefit analysis was developed for each system modernization as required by EPA's Capital Planning and Investment Control procedures, and o the cost benefit analysis contained an analysis of Exchange Network technology implementation (a key functional capability used to gain access or interface with the system). 19 ------- • Post-Implementation Reviews - We reviewed documents from the selected program offices for each respective system to determine whether: o a Post-Implementation Review was developed for each system modernization as required by Office of Management and Budget (OMB) Circular A-130, Management of Federal Information Resources, and o the Post-Implementation Review contained an analysis of Exchange Network technology implementation. • Policies and procedures to ensure program offices design their applicable systems to utilize the Exchange Network when developing new systems or when upgrading existing ones - We interviewed EPA representatives and reviewed documents to determine whether EPA had: o included the Exchange Network in the Agency's Enterprise Architecture, and o developed policies or procedures for the above objective. We interviewed program office representatives to gain an understanding of the development methodology of the cost benefit analysis documents. We interviewed Agency representatives and reviewed the Agency's current Enterprise Architecture to determine whether the Network is a part of the Enterprise Architecture. We interviewed Agency representatives to determine if policies or procedures exist to ensure program offices design their applicable systems to utilize the Network (when applicable) when developing new systems, or when upgrading existing ones. EPA Assistance to Exchange Network Governance Bodies We evaluated steps EPA could take to assist Exchange Network governance bodies (ENLC and NOB) related to (1) goal setting, (2) collaboration on Network projects, (3) solicitation of stakeholder input, (4) stakeholder participation, and (5) stakeholder representation. Specifically, we: • Reviewed ENLC and NOB charters to determine governance body objectives and purpose, • Issued questionnaires to ENLC and NOB and analyzed written responses related to the five items listed above, and • Interviewed two members of each governance body to solicit input related to the five items listed above. 20 ------- Appendix B Federal and Agency Criteria The Government Performance and Results Act of 1993 requires agencies to develop long- term strategic plans defining general goals and objectives for their programs, annual performance plans specifying measurable performance goals for all of the program activities in their budgets, and annual performance reports showing actual results compared to each annual performance goal. Section 5123 of the Information Technology Management Reform Act, commonly known as the Clinger-Cohen Act of 1996, requires the head of an executive agency to ensure that performance measurements are prescribed for information technology used by, or to be acquired for, the executive agency. These performance measurements measure how well the information technology supports programs of the executive agency. OMB Circular A-130, Management of Federal Information Resources, states as part of the evaluation component of the capital planning process that the Agency must conduct post- implementation reviews of information systems and information resource management processes. The post-implementation reviews validate estimated benefits and costs, and document effective management practices for broader use. The Agency must evaluate systems to ensure positive return on investment and decide whether continuation, modification, or termination of the systems is necessary to meet agency mission requirements. EPA Enterprise Architecture Policy, Chief Information Officer Policy Transmittal 06- 001, requires all EPA information management and technology development, modernization, enhancement, and acquisitions conform to the Enterprise Architecture and comply with applicable Enterprise Architecture requirements of the Capital Planning and Investment Control and Agency budget process, as published in periodic procedures, technical standards, and guidelines. All information management and technology development, modernization, enhancement, and acquisitions shall develop a Solution Architecture documenting the alignment of the proposed project with the Enterprise Architecture. Solution Architectures shall be certified as architecturally compliant prior to project development unless the appropriate waiver is obtained. EPA's Capital Planning and Investment Control procedures for OMB Exhibit 300 submissions require that an Alternatives Analysis with costs and benefits be completed. The document states that EPA follows OMB's guidelines and requires at least three alternatives for business case analysis, with one alternative being "as is," to continue with no change. Each viable technical approach should be included as an alternative. For the "as is" alternative, requirements will default to the current state, which will function as a baseline for the other alternatives. The 2005 Exchange Network Business Plan, endorsed by senior Agency officials, stresses the importance of developing a Network marketing plan that includes ways in which the Network adds value and that links sought-after partner activities with Network vision, goals, 21 ------- and objectives. The Network Business Plan also states the importance of implementing a "change management" process for communicating and controlling Network changes. This process helps partners initiate, sustain, and expand their use of the Network. The Network Business Plan also contains specific criteria for determining which data flows are most suitable for using the Network. These criteria include determining whether the Network will provide specific benefits by: • Making the exchange more efficient by reducing or eliminating manual intervention, such as scheduling, resubmissions, or security; and • Providing higher data quality due to additional or more efficient error checking and/or earlier detection of errors/discrepancies. The criteria also state that certain data flows are more appropriate for the Network than others. Important considerations include the extent to which data flows are likely to capture the Network effect by engaging multiple partners, as well as the volatility of data and frequency of exchange. The Network Business Plan contains, for planning purposes, a detailed description of a "full deployment" Network scenario. This scenario estimates full deployment by the year 2010 and that some of the major attributes of a fully deployed Exchange Network include: • Fifty fully functional nodes servicing 15 major flows and their associated publishing/services, • Use of the Network by nearly all States, as many tribes as possible, and several additional Federal partners, • Substantial coverage (but not necessarily full implementation) of all major regulatory program areas, and • Sufficient deployment to have begun reaping economies of scale for all Network component areas. 22 ------- Appendix C Agency Response to Draft Report June 27, 2007 MEMORANDUM SUBJECT: Office of Environmental Information Response to Draft Audit Report - Assignment No. 2006-000212 FROM: Linda A. Travers /s/ Deputy Assistant Administrator TO: Rudolph M. Brevard Director, Information Resources Management Assessments Office of Inspector General The purpose of this memorandum is to respond to your draft report of May 22, 2007, on the audit related to the National Environmental Information Exchange Network (Network). The Office of Environmental Information (OEI) concurs with all of your findings and recommendations with one exception. In many cases, we have already taken steps to implement the activities covered by your recommendations. I also want to note that the number of states and tribes using the Network, as well as the number of states using various data flows, has grown since your review. The attached appendix describes OEI's response in detail according to the terms of your cover memorandum transmitting the draft report. As you are aware, the Network is a partnership of equals with EPA, states and tribes working together to formulate strategy, establish sound governance, and ensure smooth operations. This may make the Network unique for your review in that the Agency does not manage it alone. Therefore, while we are recognized for breaking new technical ground in its development, we believe we are also breaking new ground in establishing a robust, collaborative governance structure to achieve shared Network goals with our partners. As your office moves forward with the final draft of the report, I believe it is important to infuse the findings and recommendations with an understanding of this unique and collaborative management approach. Finally, I believe this report will help us improve the Network. I look forward to sharing it with my colleagues in the EPA program offices, which I believe can help us achieve the goal of improving and expanding Network use by state and EPA environmental 23 ------- programs alike. Thank you for your team's dedicated efforts and attention. If you have any questions regarding this response, please contact me at 202-564-6665 or Mark Luttner at 566-1630. Attachment cc: Mark Luttner, OIC Myra Galbreath, OTOP Andrew Battin, OIC Doreen Sterling, OIC Bob Trent, OPRO 24 ------- APPENDIX OEI DETAILED RESPONSE TO OIG DRAFT AUDIT REPORT: IMPROVED MANAGEMENT PRACTICES NEEDED TO INCREASE USE OF THE EXCHANGE NETWORK Recommendation 2-1 The Director of the Office of Information Collection (OIC) execute recently developed Exchange Network Communications and Marketing plan elements that includes actively promoting the business value of participating in Network initiatives to EPA and partner environmental program managers. OEI Response: CONCUR The Director of OIC shall ensure recommendations are provided to the ENLC in the form of an Exchange Network Communications and Marketing plan that includes actively promoting the business value of participating in Network initiatives to EPA and partner environmental program managers. Recommendation 2-2 The Director of OIC modify Exchange Network change management policies and procedures to include step-by-step processes for fully testing and certifying all implementation tools before release to the Exchange Network community OEI Response: CONCUR The Network Technical Group (NTG), an organization of state and EPA representatives and a subgroup of the Network Operations Board (NOB) performs this function now. It will take additional steps to develop and publish these procedures to the Network Web site, www.exchangenetwork.net. Recommendation 3-1 The OEI Principal Deputy Assistant Administrator work with Exchange Network governance bodies to develop and implement a process that uses Network Business Plan criteria to evaluate data flows for future Network Implementation OEI Response: CONCUR This will be developed jointly with the Exchange Network's governance bodies. These priorities are also listed in Appendix B of the annual Exchange Network Grant Program Solicitation Notice Appendix B. 25 ------- Recommendation 4-1 The Director of OIC develop a new milestone plan for completing the EN performance measures project. OEI Response: CONCUR This has been completed. The EN performance measures project is underway and now on schedule. The draft EN measures baseline report will be ready in late July. As the Network Partnership and Resources Group (NPRG) assumes control of the project, adjustments to the overall project schedule will likely occur but mostly at the work assignment (contractor project milestone) level. Recommendation 4-2 The Deputy Assistant Administrator (DAA) of OEI work with EN governance bodies to update the charter of the NPRG to include responsibilities for overseeing the EN performance measures workgroup. OEI Response: CONCUR We believe this is already articulated clearly on Page 3 of the NPRG/NTG charter where performance measures are specifically mentioned. Recommendation 4-3 OEI DAA work with the EN governance bodies to develop and publish a charter for the EN performance measures workgroup and define roles and responsibilities for workgroup members. OEI Response: MOOT The NPRG already has already assumed responsibility for performance measures and its charter encompasses this work. The prior workgroup charged with this responsibility has been disbanded. Recommendation 4-4 OEI DAA work with the Exchange Network governance bodies to develop procedures for establishing ad-hoc workgroups for EN projects. OEI Response: CONCUR This will be implemented with any new work group. Recommendation 5-1 Director, Office of Technology, Operations and Planning (OTOP) include steps to analyze Exchange network technologies as part of the cost benefit analysis process in the Capital Planning and Investment Control (CPIC) procedures OEI Response: NON-CONCUR The cost benefit analysis process in the Agency's CPIC procedures is intended to be a high level "how-to" set of guidelines. It is not scoped at the level of detail needed to include such an analysis as the recommendation suggests. The CPIC major IT investment review process covers fewer than two dozen systems, and not all of the CPIC major investments are candidates for 26 ------- using the network (e.g., Technology Infrastructure Modernization). Thus, this is better left as a case-by-case architectural implementation decision for each solution. This decision must balance the Agency's needed solution architecture tools and methods with program needs and budget constraints. Recommendation 5-2 Director, OTOP, should include the EN and related technologies in the Agency's enterprise architecture OEI Response: CONCUR EPA's information exchange technologies associated with the Exchange Network and related technologies are part of the Agency's Enterprise Architecture, In addition, EPA's architectural development process, as part of the new System Life Cycle Management procedures, specifies that a solution architecture be documented early on in the life cycle development process of an IT project. This provides ample opportunity to identify which IT projects needing information collection functions are appropriately planning to use the technologies associated with the Exchange Network. Rules for use of the Exchange Network are part of a broader set of rules that OEI must issue (after Agency review) to govern/guide the use of enterprise tools. These rules (solutions architecture standards/guidance) have recently been under discussion within OEI, and a management team has been formed to start the development process. Recommendation 5-3 Director, OIC, should develop a policy and procedure to guide program offices to use EN technologies, when applicable, when developing new systems or modifying existing systems OEI Response: CONCUR The Director of OIC will ensure development of CDX procedures focused on EPA Program Office technology integration with the EN when developing new systems or modifying existing systems where applicable by Q4 2008. In addition, the ENLC (Exchange Network Leadership Council comprised of State and EPA executives) shall develop an EN Interoperability Policy focused on state and EPA integration with the EN where applicable by Q4 2008. 27 ------- Appendix D Distribution Office of the Administrator Office of General Counsel Assistant Administrator for Environmental Information and Chief Information Officer Deputy Assistant Administrator for Environmental Information Director, Office of Technology Operations and Planning, Office of Environmental Information Director, Office of Information Collection, Office of Environmental Information Deputy Director, Office of Information Collection, Office of Environmental Information Audit Coordinator, Office of Environmental Information Agency Followup Official (the CFO) Agency Followup Coordinator Associate Administrator for Congressional and Intergovernmental Relations Associate Administrator for Public Affairs Acting Inspector General 28 ------- |