vvEPA U.S. Environmental Protection Agency DATA CENTER OPTIMIZATION INITIATIVE STRATEGIC PLAN April Page 1 ------- EPA Data Center Consolidation Plan TABLE OF CONTENTS 1 Executive Summary 3 2 Background 4 3 Introduction 4 4 Data Center Consolidation and Closure Targets 5 5 Cost Savings 7 6 Data Center Optimization Metrics 8 7 Conclusion 12 Page 2 ------- EPA Data Center Consolidation Plan 1 Executive Summary In 2016, the Office of Management and Budget (OMB) launched the Data Center Optimization Initiative (DCOI) as a successor to the Federal Data Center Consolidation Initiative (FDCCI). On August 1st, 2016, the Federal Chief Information Officer (CIO) issued memorandum M-16-19 which provided the framework for agencies to achieve data center consolidation and optimization requirements. Under memorandum M-16-19 agencies are required to develop and report on data center strategies to consolidate inefficient infrastructure, optimize existing facilities, improve security posture, achieve cost savings, and transition to more efficient infrastructures, such as cloud services and interagency shared services. In addition, OMB set a closure goal for the U.S. Environmental Protection Agency (EPA) of 34 non-tiered data centers by Fiscal Year 2018. As of April 2018, the EPA has completed 19 data center closures, maintained energy metering at 100% for tiered data centers, maintained a power usage effectiveness (PUE) of 1.5 or below and a virtualization ratio over 4. The EPA continues to work towards additional data center closures and anticipates having 22 total closures by the end of September 2018. As of March 2018, all EPA tiered data centers have energy metering installed, yet lack automated reporting capability as required in M-16-19. The EPA is evaluating data center information management (DCIM) tools to help accomplish this task. The EPA is also working to expand the utilization of its ScienceLogic (EM7) tool to identify areas of improvement with automated monitoring of server utilization and virtualization. Agency facility utilization remains around 34%. The EPA is pursuing interagency hosting agreements with other agencies to fill the 14,000 sq. ft. data center in Research Triangle Park and has provided space for colocation of OMB and GSA data center assets. EPA's Data Center Infrastructure Management (DCIM) tool deployment and the assignment of a Data Center Energy Practitioner (DCEP) for each data center has been extended into FY 2019 due to budget uncertainties. Page 3 ------- EPA Data Center Consolidation Plan 2 Background In August 2016, the Office of Management and Budget (OMB) released memorandum M-16-19 that established the Data Center Optimization Initiative (DCOI) which superseded the previous Federal Data Center Consolidation Initiative (FDCCI). DCOI's consolidation and optimization requirements have a targeted completion date of September 2018 and are summarized as follows: • Meet data center closure and optimization targets and reassess cost savings. • Assign certified Data Center Energy Practitioners (DCEP) for each data center. • Implement automated energy metering and Data Center Infrastructure Management (DCIM) tools at all tiered data centers. • On a quarterly basis, report to OMB Agency progress with respect to data center closure/consolidation targets, optimization metrics, and cost savings and avoidance targets. • Update DCOI FITARA milestones on a quarterly basis. 3 Introduction Consolidation and optimization of federal data centers has been a priority for the EPA. At the inception of DCOI, EPA had 58 non-tiered data centers and computer rooms that supported business operations for EPA facilities located across the country. Per OMB criteria, four of EPA's data centers are classified as tiered data centers. These include the following: • National Computer Center (NCC) in Research Triangle Park, North Carolina. • Potomac Yards Data Center in Washington, DC. • Region 5 Data Center in Chicago, Illinois. • Region 8 Data Center in Denver Colorado. To improve efficiency and consistency of data center operations, EPA implemented an Agency- wide Physical-to-Virtual (P2V) policy requiring offices to convert existing physical servers to virtual servers wherever possible. EPA also defined server and software standards for virtualized platforms and established an enterprise platform for infrastructure monitoring. EPA implemented strategic sourcing initiatives to pool resources and negotiate optimal pricing for IT products and services. EPA is pursuing opportunities to establish shared data center services and encourage consolidation of data center functions. EPA established centralized resources for continuity of operations planning (COOP) and disaster recovery (DR) in the tiered data centers. Region 6 and region 9 are currently leveraging these services and the Region 4 implementation is being operationalized. The General Services Administration (GSA) recently established a five-year agreement with EPA to leverage NCC facilities and hosting services. To meet DCOI reporting requirements, EPA is in the process of performing market research on Data Center Infrastructure Management (DCIM) tools. None of EPA's tiered data centers currently have automated monitoring software for the collection of PUE. EPA budget uncertainties have led to necessary schedule adjustments for testing and procurement of a DCIM Page 4 ------- EPA Data Center Consolidation Plan solution. DCIM tool procurement is anticipated in FY 2019. For non-tiered data centers, EPA plans to leverage its existing EM7 tool to support system monitoring requirements of server utilization and virtualization. EPA estimates that it is currently at a Power Usage Effectiveness (PUE) ratio of 1.5 which is within the requirements for DCOI. EPA's efforts to consolidate and optimize its data centers have been significant. However, it is important to acknowledge that these efforts involved considerable resource commitments to upgrade data center infrastructure, consolidate data center assets, and perform tool assessments. We are confident that these efforts will enable EPA to better monitor and manage its energy consumption and report on OMB's optimization metrics, but they may not produce cost savings in the near-term. Additionally, EPA must also balance consolidation efforts with network costs and application performance requirements. For example, some EPA regional offices, research centers, labs and other facilities host local infrastructure such as telecommunications infrastructure, specialized lab and research systems, and COOP and DR. The distributed nature of this infrastructure and the need to support these critical requirements can make further consolidation challenging. As EPA moves forward to achieve DCOI requirements, it will need to continue to balance the benefits of consolidation with operational requirements and costs. The remainder of this document provides more detail on EPA's progress and plans for meeting DCOI requirements. 4 Data Center Consolidation and Closure Targets Current Progress OMB assigned a target goal for the EPA to close 34 non-tiered data centers by September 2018. At the inception of the DCOI, EPA maintained 58 telecommunication closets and computer rooms that met the classification of a data center per memorandum M-16-19. These data centers support business operations for EPA facilities including Headquarters (DC Metro area), Research Triangle Park (RTP), 10 regional offices and 13 HQ Program Offices. The remaining facilities are small field offices and continuity of operations (COOP) sites. Per OMB's criteria, EPA classifies four of its data centers as tiered. The tiered data centers support EPA's enterprise information technology (IT) operations. EPA's primary data center, is the NCC, located in RTP, North Carolina. Figure 1 depicts EPA's tiered data centers, computer rooms and network topology. EPA is currently in the process of downgrading the Region 8, tiered data center and replacing the Potomac Yards data center with the National Enforcement Investigations Center (NEIC) which will serve as EPA's western presence data center. At project completion, EPA will have three tiered data centers. All tiered data centers will be operated in a standardized fashion and will implement configurations that maximize power and cooling efficiency. Page 5 ------- EPA Data Center Consolidation Plan Figure 1 - EPA's Primary Data Centers PRIMARY DATA CENTER Regional Field Offices Regional HQ COOP Sites EPA Labs To meet OMB's closure target, the EPA reviewed its existing data centers and revised its approach for consolidation utilized under FDCCI. For geographic areas where EPA has multiple data centers, a single facility was designated into which data center IT assets will be consolidated. Additionally, the EPA intends to consolidate all non-tiered data centers not designated to be retained to serve a geographic area unless it is proven infeasible to close some data centers due to technical limitations that compromise agency mission essential functions. Through these efforts, EPA has identified potential closures sufficient to meet OMB's target. Several validation activities are required before confirming this total. As Table 1 shows, under FDCCI, EPA closed 21 of its data centers. For DCOI, EPA has closed 19 data centers, 3 are in progress and expect to complete prior to the 2018 deadline. Table 1 - Completed and Planned Data Center Closures by Fiscal Year Completed and Planned Data Center Closures by Fiscal Year FY 2010 FY 2011 FY 2012 FY 2013 FY 2014 Q.l/2 FY 2015 03/4 FY 2015 FY 2016 FY 2017 Q1 FY 2018 Q2 FY 2018 03 FY 2018 04 FY 2018 0 1 14 3 1 2 3 3 12 1 0 3 0 21 Total FDCCI Closures 19 DCOI Closures Completed 3 Planned 21 Total FDCCI 22 Total DCOI Page 6 ------- EPA Data Center Consolidation Plan Activities Underway For those data centers targeted for closure, DCOI points of contact (POCs) are reviewing their data center IT assets and determining which to decommission, consolidate and/or virtualize. They will then execute consolidation plans to move applicable data center IT assets to the Cloud, an EPA tiered data center or the data center retained for that geographic area. The final phase of the effort will require DCOI POCs to surplus or excess data center IT assets and scale remaining data center assets to align with reduced capacity requirements. Efforts are also underway to align additional closures with lease timelines and funding availability. Challenges and Risks In some cases, EPA's regional offices, research centers, labs and field offices host local infrastructure data center assets such as specialized lab and research support, emergency response (ER) and COOP that may have non-severable configurations and potentially, must remain co- located to operate and fulfill mission functions. The distributed nature of these offices and the continuation of required localized mission functions require EPA to balance consolidation efforts with continuity of business functions, application performance requirements, cost and security. Testing is required to validate operational functionality before migrating these systems away from the local site. EPA has identified several laboratories with servers running Laboratory Information Management Systems (LIMS) that interface with LAN-based scientific equipment. To migrate the LIMS servers, the EPA is required to test the WAN capabilities of these systems as well as procure network and security hardware to provide additional security as the scientific equipment is isolated due to security concerns. Budgetary uncertainties have extended execution of efforts required to perform validation and procurement activities into FY 2019. 5 Cost Savings Given the upfront investments for infrastructure upgrades, tool assessment and validation, the EPA does not expect significant near-term cost savings under DCOI. The EPA structured its data center consolidation work to promote cost savings in areas of increased facility utilization, reduced energy consumption, maximized server and storage use, and reduction in the long-term growth of IT infrastructure costs. EPA is continuing to refine and enhance DCOI cost savings calculations to ensure accuracy. Though OMB agreed there would be negligible savings in the near-term, the EPA has begun gathering additional information to track any realized costs or savings within our quarterly progress data calls. Projected costs needed to fulfill identified DCOI goals include planning and design activities, updates to data center facilities (e.g., uninterruptable power supply, power distribution unit, and computer room cooling equipment), validation testing of local applications across the Wide Area Network (WAN), DCIM tool and security device procurements, and DCEP training and/or hiring. Additionally, the EPA required investment in WAN bandwidth upgrades to accommodate additional traffic that is no longer isolated to the Local Area Network (LAN) infrastructure. These investments, while critical to support long-term DCOI objectives, offset any near -term cost savings. In the longer-term, however, the Agency expects these improvements will reduce energy consumption leading to greater efficiency and demonstrable cost reductions. Page 7 ------- EPA Data Center Consolidation Plan 6 Data Center Optimization Metrics Current Progress OMB established several optimization metrics under DCOI to include: energy metering, Power Usage Effectiveness (PUE), facility utilization, as well as automated monitoring of server utilization and virtualization. All EPA tiered data centers have some form of energy metering installed. While current metering is adequate to provide an estimated amount of energy consumption to calculate PUE, submetering at some of the tiered data centers is required for precise PUE calculations. The EPA currently reports a PUE of 1.5 and does not expect to implement any additional submetering at any of the remaining data centers. M-16-19 requires federal agencies to assign a DCEP to all data centers. At the outset of DCOI, the EPA did not have a certified DCEP on-staff. OEI will fulfill this requirement in one of two ways: for the near term, existing IT services contracts will be assessed to determine the feasibility of using contractor resources to provide the role of a DCEP for its data centers. For the longer term, OEI will consider utilizing EPA staff in the Office of Administration and Resources Management (OARM) as they currently have the responsibility for provisioning facilities management and services. OEI will also give due consideration to creating a federal position and recruiting a DCEP candidate. Under DCOI, federal agencies strive to maintain an 80% facility utilization calculated using a portion of total gross floor area in tiered data centers that is actively utilized for racks that contain IT equipment. The EPA maintains the National Computer Center (NCC) in Research Triangle Park (RTP) as its primary data center. This facility is owned by EPA. This 14,000+ sq. ft. facility is utilized for regional COOP, application hosting and development, and as a co-location hosting facility for other federal agencies such as GSA. Through consolidation and optimization of technologies, much of the utilized floor space in the NCC was emptied. While these efforts increased virtualization and energy efficiency numbers, they caused facility utilization to drop. The EPA is working to advertise co-location hosting services to other federal agencies to leverage the NCC facility and meet DCOI facility utilization requirements. Due to the amount of available floor space, the amount of additional equipment that is required to fill the facility, as well as other optimization efforts that are in direct conflict with the facility utilization metric, the EPA expects to meet the facility utilization metric beyond OMB's FY 2018 deadline. M-16-19 requires agencies to procure and implement a DCIM tool at all tiered data centers. The EPA looked at several options and due to budgetary uncertainties, elected to delay procuring a commercial solution as required under DCOI. Based on feedback from other federal agencies within the DCOI communities of practice meetings, the lead time for full implementation of a DCIM tool will place it outside the DCOI implementation timeline. The EPA is continuing to identify options that may help satisfy DCOI requirements. OMB required federal agencies to implement automated monitoring to provide at minimum: server utilization and virtualization ratios. The EPA has determined that a tool for which the Agency already has a license, EM7, will provide automated capabilities outlined within DCOI and has already begun the process of upgrading the licensing and collection capabilities to expand to all Page 8 ------- EPA Data Center Consolidation Plan agency servers. The Office of Environmental Information issued an internal memorandum requiring all agency servers to be configured for data collection within EM7. The EPA expects to have all agency servers configured within EM7 by the end of CY 2018. The EPA also started work on a non-standard report based on DCOI formulas for server utilization. The required DCOI server utilization report has proven challenging as there is no industry standard report that can be leveraged and a custom report must be created. Additionally, multiple reports must be made to properly cover the different types of servers and reduce the number of "false positives" that would be generated from a single uniform report. EPA expects to be able to implement automated server monitoring by the DCOI deadline. However, further enhancements will be needed to provide the level of specificity required under DCOI. The EPA has made considerable progress optimizing its data centers and utilizing consolidation. These improvements enable EPA to meet some of the DCOI targets. For example, EPA implemented a "virtual first" policy requiring virtual servers be deployed by default for all server instances. At the NCC, the EPA initiated a phased data center modernization program which included a redesigned layout for more efficient use of floor space, turning off air handlers, raising the operating temperature towards the upper end of manufacturers' specifications, and preliminary implementation of cold aisle containment technologies. Similar improvements are planned for the other tiered data centers. Several program and regional offices pursued similar virtualization initiatives, implemented more efficient LED lighting, and/or made improvements to server room cooling systems to reduce energy consumption. For example, Regions 9 and 10 implemented airside economizers and cold aisle containment systems reducing energy consumption over traditional HVAC systems. Region 9's improvements included raising the operating temperature to minimize energy consumption. The result of these improvements allowed Region 10 to achieve a 47% reduction using the airside economizer as compared to prior year HVAC energy consumption. Region 10 estimates that over a 10-year lease, the Region will save at least 915,782 kWh and approximately $89,747 (using 9.8 cents per kWh for Seattle, WA). EPA also completed the following activities to improve efficiency and encourage consolidation: • Standardization and Enterprise Procurement: EPA established server and software standards for the x86/64 virtualization platforms and established an enterprise platform for infrastructure monitoring. The Agency will continue to roll out these standards in upcoming years and verify progress toward achieving the targeted optimization metrics. EPA also implemented multiple strategic sourcing initiatives to negotiate optimal pricing for IT products and services. EPA reviews all IT procurements as part of the FITARA process thereby ensuring organizations are aware of and adhere to Federal and Agency initiatives and remain aligned with EPA's overall strategic direction. • Enterprise COOP and Disaster Recovery (DR): Many EPA offices currently provide COOP and DR services using site-specific solutions. EPA established centralized resources for COOP and DR in the tiered data centers to enable the Agency to offer shared service options and encourage consolidation. Enterprise COOP and DR services provide remotely accessible data and applications to support continuity of operations and disaster recovery to EPA regions or field offices. EPA successfully implemented Page 9 ------- EPA Data Center Consolidation Plan virtualized COOP for Region 6 and 9 and is looking to implement a similar configuration for the other regions. • Data Center Hosting Services - The EPA entered into an agreement with GSA to provide data center hosting services at the NCC. This engagement began in September 2016 as a phased deployment that allows GSA to strategically optimize its data center footprint and reduce operating costs over the next several years. The initial five-year agreement provides GSA with facility hosting space and associated environmental requirements (air conditioning and humidity control, electrical, physical security, etc.) as well as provisions for executing options to continue hosting at the NCC. EPA plans to develop documentation and procedures to enable the Agency to standardize and reuse its processes for future co-location hosting engagements with other federal organizations. Moving Forward For the remaining data centers, data center POCs are working with EPA management and data center staff to identify opportunities for greater efficiencies, make necessary upgrades to address any facilities, network capacity or operational issues, decommission and excess unneeded IT assets, and appropriately scale remaining assets to align with reduced capacity requirements. EPA performed market research on DCIM tools to support automated collection and reporting of data center hardware/software inventories and optimization metrics in its tiered data centers. EPA has identified options for DCIM software, procurement and implementation. Open source solutions such as OpenDCIM are also being evaluated. The procurement and implementation an agency DCIM solution is anticipated by the end of FY 2019. For non-tiered data centers, EPA has chosen to expand its current EM7 tool for implementing automated system monitoring. OEI has directed internal organizations to register all servers into EM7. The EPA plans to utilize EM7 monitoring to identify those data centers and servers not meeting efficiency standards, address deficiencies, and meet power efficiency goals. EPA estimates that it is currently at a Power Usage Effectiveness (PUE) ratio of 1.5. The Agency anticipates that virtualization and planned optimization activities will enable the Agency to maintain a PUE of 1.5 through FY 2018. Table 2 outlines planned and achieved performance levels for each optimization metric for the 2018 fiscal year. Please note: The estimated figure for the Server Utilization and Automated Monitoring metric is due to lack of automated software available to provide current percentages. Once EM7 is deployed and an adequate report generated, the EPA will be able to report server utilization numbers and will work to begin optimization of unused compute resources. Page 10 ------- EPA Data Center Consolidation Plan Table 2 - Planned and Achieved Performance Levels for Optimization Metrics No. Optimization Metric FY 2018 Target Value Current Value (Estimated) Planned FY 2018 1 Energy Metering 100% 100% 100% 2 Power Usage Effectiveness (PUE) < 1.5 1.5 1.5 3 Virtualization >4 6.25 >4 4 Server Utilization & Automated Monitoring >65% 0% 20% 5 Facility Utilization >80% 34% 35% Challenges and Risks EPA has faced budgetary constraints and procurement delays which have hindered the migration of IT systems to centralized data centers. For example, new procurement requirements resulting from a protest decision have delayed acquisition of high capacity networking devices which slowed migration of systems into the centralized data centers. The EPA maintains many laboratory environments that leverage a Laboratory Information Management System (LIMS) to record, manage and store data from lab instruments. The EPA has looked into options to migrate LIMS servers away from the lab environments as several are the last server remaining to consider the data center closed. LIMS proves challenging in that there is significant cost to migrate the systems. First, many of the LIMS systems reside on isolated networks that will require security hardware to allow for NIST-compliant transmission of data. Next, bandwidth increases are necessary to accommodate the sending of data across the WAN. There are also costs associated with piloting multiple vendor solutions in WAN-based configurations. Consolidating all LIMS to a single vendor would incur significant costs as databases will need to be modified and data adjusted to ensure consistency as well as training and additional testing for equipment compatibility. The EPA plans to continue investigating options to migrate the LIMS servers from the smaller laboratories, but due to costs, complexity and timeframes is unable to complete within the timeline of DCOI. The increased accuracy of optimization metrics and automated reporting of OMB's optimization metrics is dependent on the procurement and implementation of a DCIM tool and upgrades of the EM7 tool. While the Agency is working to plan for and mitigate potential procurement and implementation risks, delays in either process could further impact EPA's ability to meet DCOI targets. As automation tools are installed and configured, the EPA anticipates that future utilization and PUE numbers may differ from those previously reported due to direct measurement at the source provided by these specialized tools. Additionally, the EPA is still working towards refining and documenting server metrics, idle Page 11 ------- EPA Data Center Consolidation Plan thresholds and specific server properties for accurately reporting server utilization between different server types. Investments are also required to support the planning, design and implementation of optimization activities. These investments are not insignificant and may make it difficult for the Agency to demonstrate cost savings. Schedule adjustments from realized budgetary uncertainties have also delayed the implementation of DCOI. 7 Conclusion Significant progress has been made to optimize and consolidate EPA data centers. The remaining planned improvements will enable the Agency to meet some, but not all of OMB's DCOI objectives. Challenges include technical issues, ensuring business needs are met, and resource constraints. Investing in further consolidation will enable the agency to better monitor and reduce energy consumption, but may not produce cost savings in the near-term. EPA must balance consolidation efforts with network costs and application performance requirements. Because some EPA regional offices, research centers, labs and other facilities host local infrastructure such as, telecommunications infrastructure, specialized lab and research systems, and COOP and DR in their data centers, further consolidation can be challenging. As EPA moves forward to achieve DCOI requirements, it will need to continue to balance the benefits of consolidation with operational requirements and implementation costs. Page 12 ------- |