^tD sr4^ # JL \ KSBZ! U.S. ENVIRONMENTAL PROTECTION AGENCY OFFICE OF INSPECTOR GENERAL Weaknesses in EPA's Management of the Radiation Network System Demand Attention Report No. 12-P-0417 April 19, 2012 Scan this mobile code to learn more about the EPA OIG. ------- Report Contributors: Richard Eyermann Mike Davis Jim Haller Marcia Hirt-Reigeluth Jennifer Hutkoff Yeon Kim Heather Layne Abbreviations CIPP Critical Infrastructure and Key Resources Protection Plan CO Contracting officer COR Contracting officer representative CPARS Contractor Performance Assessment Reporting System CPS Contractor Performance System EPA U.S. Environmental Protection Agency EPAAR Environmental Protection Agency Acquisition Regulations FAR Federal Acquisition Regulation GFP Government-furnished property MATS Management Audit Tracking System MPR Monthly progress report NAREL National Air and Radiation Environmental Laboratory NIH National Institutes of Health OAM Office of Acquisition Management OAR Office of Air and Radiation OIG Office of Inspector General ORIA Office of Radiation and Indoor Air PPIRS Past Performance Information Retrieval System QAPP Quality Assurance Project Plan RadNet Radiation Network SOW Statement of work Cover photo: Stationary RadNet monitor. (EPA photo) Hotline To report fraud, waste, or abuse, contact us through one of the following methods: e-mail: OIG Hotiirie@epa.aov write: EPA Inspector General Hotline phone: 1-888-546-8740 1200 Pennsylvania Avenue NW fax: 202-566-2599 Mailcode 2431T online: http://www.epa.gov/oig/hotline.htm Washington, DC 20460 ------- ^tDsx £ A \ V PRO^SO U.S. Environmental Protection Agency Office of Inspector General At a Glance 12-P-0417 April 19, 2012 Why We Did This Review The U.S. Environmental Protection Agency (EPA) Office of Inspector General (OIG) sought to determine whether EPA is following quality control procedures to ensure that data submitted from Radiation Network (RadNet) monitors nationwide are reliable and accurate, and whether EPA effectively implemented corrective actions in response to the EPA OIG's January 27, 2009, audit report on RadNet. Background EPA's December 2004 Critical Infrastructure and Key Resources Protection Plan identified RadNet monitors as critical infrastructure. The mission of RadNet is to monitor environmental radioactivity in the United States to provide high-quality data for assessing public exposure and environmental impacts resulting from nuclear emergencies, and to provide baseline data during routine conditions. RadNet played a critical role in monitoring radiation levels in the United States during the March 2011 Japan nuclear incident. For further information, contact our Office of Congressional and Public Affairs at (202) 566-2391. The full report is at: www.epa.aov/oia/reports/2012/ 20120419-12-P-0417.pdf Weaknesses in EPA's Management of the Radiation Network System Demand Attention What We Found Broken RadNet monitors and late filter changes impaired this critical infrastructure asset. On March 11, 2011, at the time of the Japan nuclear incident, 25 of the 124 installed RadNet monitors, or 20 percent, were out of service for an average of 130 days. The service contractor completed repairs for all monitors by April 8, 2011. In addition, 6 of the 12 RadNet monitors we sampled had gone over 8 weeks without a filter change, and 2 of those for over 300 days. Because EPA managed RadNet with lower than required priority, parts shortages and insufficient contract oversight contributed to extensive delays in fixing broken monitors. In addition, broken RadNet monitors and relaxed quality controls contributed to the filters not being changed timely. Out-of-service monitors and unchanged filters may reduce the quality and availability of critical data needed to assess radioactive threats to public health and the environment. EPA remains behind schedule for installing the RadNet monitors and did not fully resolve contracting issues identified in the OIG's January 2009 report. Until EPA improves contractor oversight, the Agency's ability to use RadNet data to protect human health and the environment, and meet requirements established in the National Response Framework for Nuclear Radiological Incidents, is potentially impaired. What We Recommend We recommend that the Assistant Administrator for Air and Radiation establish and enforce expectations for RadNet operations readiness. We recommend improved planning and management of parts availability, monitoring of filter replacement and operators, and monitoring of the installation of the remaining RadNet monitors. Further, we recommend that the Assistant Administrator, in conjunction with the Assistant Administrator for Administration and Resources Management, hold contractors accountable by establishing milestones, using incentives and disincentives, requiring contracting officers and contracting officers' representatives to formally evaluate RadNet contractors annually, and ensure that the Agency's Management Audit Tracking System is accurate and current. The Agency concurred with the recommendations except for developing metrics for evaluating frequency of filter changes and completing contractor performance evaluations, which is considered unresolved. The Agency also proposed revised language, which we incorporated where appropriate. ------- ^1tDsrx ,2& \ UNITED STATES ENVIRONMENTAL PROTECTION AGENCY ° VlV * VPB0^° WASHINGTON, D.C. 20460 THE INSPECTOR GENERAL April 19, 2012 MEMORANDUM SUBJECT: Weaknesses in EPA's Management of the Radiation Network System Demand Attention Report No. 12-P-0417 FROM: TO: Craig E. Hooks Assistant Administrator for Administration and Resources Management Arthur A. Elkins, Jr. Gina McCarthy Assistant Administrator for Air and Radiation This is our report on the subject audit conducted by the Office of Inspector General (OIG) of the U.S. Environmental Protection Agency (EPA). This report contains findings that describe the problems the OIG has identified and corrective actions the OIG recommends. This report represents the opinion of the OIG and does not necessarily represent the final EPA position. Final determinations on matters in this report will be made by EPA managers in accordance with established audit resolution procedures. Action Required In accordance with EPA Manual 2750, you are required to provide a written response to this report within 90 calendar days. Your response will be posted on the OIG's public website, along with our comments on your response. Your response should be provided in an Adobe PDF file that complies with the accessibility requirements of Section 508 of the Rehabilitation Act of 1973, as amended. If your response contains data that you do not want to be released to the public, you should identify the data for redaction. You should include a corrective actions plan for agreed-upon actions, including milestone dates. We have no objections to the further release of this report to the public. This report will be available at http ://www. epa. gov/oi g. If you or your staff have any questions regarding this report, please contact Melissa Heist, Assistant Inspector General for Audit, at (202) 566- 0899 heist.melissa@epa. gov; or Mike Davis, Acting Director for Efficiency Audits, at (513) 487-2363 or davis.michaeld@,epa. gov. ------- Weaknesses in EPA's Management of the Radiation Network System Demand Attention 12-P-0417 Table of Contents Chapters 1 Introduction 1 Purpose 1 Background 1 Prior EPA OIG Reports 4 Noteworthy Achievements 5 Scope and Methodology 5 2 EPA Not Managing RadNet as a High-Priority Program 7 Contract and QAPP Define Repair and Filter Change Time Frames 7 One in Five RadNet Monitors Out of Service an Average of 130 Days 7 Filters for RadNet Monitors Not Changed Timely 10 Limited Resources and Competing Priorities Impacted RadNet Readiness 12 Conclusion 13 Recommendations 14 Agency Response and OIG Evaluation 14 3 EPA Should Improve RadNet Contract Oversight 16 Federal and EPA Guidance Define Contract Management Requirements 16 Incentives/Disincentives Not Included in All RadNet Contracts 17 Installation of RadNet Monitors Behind Schedule 19 Monthly Progress Reports Not Effectively Used to Manage RadNet Contracts 21 Contractor Performance Evaluations Not Timely or Completed 22 Conclusion 23 Recommendations 24 Agency Response and OIG Evaluation 24 Status of Recommendations and Potential Monetary Benefits 26 Appendices A Analysis of Prior EPA OIG Report Recommendations 28 B Analysis of 25 Out-of-Service Monitors as of March 11, 2011— Date of Japan Nuclear Incident 30 -continued- ------- Weaknesses in EPA's Management of the Radiation Network System Demand Attention 12-P-0417 C Extensions to the Fixed Air Monitoring Stations Contract and EPA Compensation Received 31 D Agency Response to Draft Report 32 E Distribution 44 ------- Chapter 1 Introduction Purpose In January 2009, the Office of Inspector General (OIG) issued an audit report that contained recommendations for improving the management and oversight of the U.S. Environmental Protection Agency's (EPA's) Radiation Network (RadNet) system. The nature and importance of the prior report recommendations and the recent attention on RadNet due to the critical role it played in the United States during the March 2011 Japan nuclear incident warranted a follow-up audit. Our audit objectives were to determine: • Whether EPA is following quality control procedures to ensure that data submitted from the RadNet monitors nationwide are reliable and accurate • Whether EPA effectively implemented corrective actions in response to our January 27, 2009, audit report, EPA Plans for Managing Counter Terrorism/Emergency Response Equipment and Protecting Critical Assets Not Fully Implemented Background The Patriot Act of 2001 defined critical infrastructure as assets so vital to the United States that their incapacity or destruction would have a debilitating impact on public health or safety. Homeland Security Presidential Directive No. 7, December 17, 2003, required federal agencies to identify, prioritize, and protect Critical Infrastructure and Key Resources Protection Plan (CIPP) assets. EPA's December 2004 CIPP identified RadNet monitors as critical infrastructure. The June 2008 Nuclear/Radiological Incident .Annex to the National Response Framework lists EPA and RadNet as a key federal radiological resource and asset. EPA's RadNet System RadNet, a national network of monitoring stations, provides real-time monitoring of environmental levels of radiation in the United States. Monitoring stations regularly collect air, precipitation, drinking water, and milk samples for analysis of radioactivity. RadNet has three objectives: Stationary RadNet monitor. (EPA OIG photo) 12-P-0417 1 ------- • Provide data for nuclear emergency response assessments • Provide data on ambient levels of radiation in the environment for baseline and trend analysis • Inform the general public and public officials about radiation levels EPA's RadNet system consists of 124 stationary (fixed) monitors1 and 40 deployable air monitors that can be sent to take readings anywhere in the country (figure 1). Our audit focused on EPA's stationary RadNet air monitoring system. Figure 1: Locations of RadNet monitors nationwide as of April 2011 wa « MT MA MN OR ao PA NE -\D£ MOl UT KS. KY DC t)K AZ sc NM LA j TX r~vn American Samoa Northern Mariana Islands o Source: EPA Japan Nuclear Emergency: Radiation Monitoring website, http: //www, epa.gov/iapan2011/rert/radnet-data-ma p.htm I/. 1 Stationary monitors collect information on air particulates. When fully implemented, the RadNet system will consist of 134 stationary monitors. The remaining 10 monitors are planned to be installed, by June 2012, in Boise City, Idaho; Casper, Wyoming; Champaign, Illinois; Charleston, South Carolina; Great Falls, Montana; Greensboro, North Carolina; LaCrosse, Wisconsin; Scranton, Pennsylvania; Shawano, Wisconsin; and St. George, Utah. 12-P-0417 2 ------- RadNet monitors measure airborne radiation collected on filters 24 hours a day, 7 days a week, and submit data for analysis to the EPA National Air and Radiation Environmental Laboratory (NAREL) in Montgomery, Alabama. NAREL is part of the Office of Air and Radiation (OAR), Office of Radiation and Indoor Air (ORIA). These data allow NAREL to identify normal background radiation levels in an area. A computer continually reviews the real-time air monitoring data and, if the results show an increase in radiation levels outside of the normal range, the computer immediately alerts EPA laboratory staff so they can review the data to ensure accuracy. Operators of the fixed monitors also send filters to NAREL for further analysis. The detailed filter analysis allows NAREL to see the trace amounts of radioactive material that the real-time air monitors do not pick up. NAREL worked with the EPA Office of Acquisition Management (OAM) to award the three contracts that are the subject of this review—one each for the manufacture of stationary air monitors, repair and maintenance services, and ordering of needed spare and repair parts. These contracts are crucial to the success of the RadNet program. OAM awarded the three contracts from 2007 to 2010, and they have a total contract obligation value of over $8 million (table 1). The contracts for fixed air monitoring stations and spare parts were awarded to the same contractor due to the proprietary nature of those spare parts. Table 1: RadNet contracts covered by this review RadNet contract no. Purpose of contract Award date Total contract obligations EP-W-07-076 Fixed air monitoring stations 09/28/2007 $5,326,210 EP-D-08-068 Repair and maintenance services 05/05/2008 1,489,880 EP-D-10-085 Spare parts 09/21/2010 1,405,913 Total $8,222,003 Source: EPA Active Contract Listing and Financial Data Warehouse as of February 9, 2012. Japan Nuclear Incident On March 11, 2011, the magnitude 9.0 Tohoku earthquake in northern Japan created a tsunami that damaged the Fukushima nuclear power plant. In response to the Japan nuclear incident, EPA increased sampling frequency and analysis to detect and measure radiation levels, and inform the public of any changes in those levels. EPA also increased sampling frequency for the milk and drinking water networks, and increased analysis frequency for all networks to detect and measure radiation levels. On April 2, 2011, an EPA press release stated that several EPA air monitors detected very low levels of radioactive material in the United States consistent with estimates from the damaged nuclear reactors. EPA explained that these detections were expected, and the levels detected were far below levels of public health concern. 2 OAM awarded earlier RadNet contracts; those contracts are not part of this review. 12-P-0417 3 ------- To provide additional geographic coverage to areas in close proximity to the releases in Japan, EPA shipped deployable monitors to islands in the Pacific, including Guam and the Commonwealth of the Northern Mariana Islands, and to locations in the western United States, including Hawaii, Idaho, and Alaska. On April 12, 2011, the U.S. Senate Committee on Environment and Public Works held a hearing, "Review of the Nuclear Emergency in Japan and Implications for the U.S.," and the EPA Administrator testified on the systems EPA has in place to protect the American public and environment. Since the events in Japan occurred, EPA's website has had thousands of views and we have received many positive comments from the public on the information we have made available. The Agency will continue to provide [RadNetJ monitoring results to the public in a very open and transparent manner. While we do not expect radiation from the damaged Japanese reactors to reach the United States at harmful levels, I want to assure you that EPA will continue our coordination with our federal partners to monitor the air, milk, precipitation and drinking water for any changes, and we will continue our outreach to the public and the elected officials to provide information on our monitoring results. Lisa Jackson, EPA Administrator, Testimony Before the United States Senate, Committee on Environment and Public Works, April 12, 2011. On May 3, 2011, EPA announced that after a thorough data review showing declining radiation levels related to the Japan nuclear incident, it had returned to the routine RadNet sampling and analysis processes. Prior EPA OIG Reports On April 26, 2006, the OIG issued Report No. 2006-P-00022, EPA Needs to Better Implement Plan for Protecting Critical Infrastructure and Key Resources Used to Respond to Terrorist Attacks and Disasters. We reported that EPA listed RadNet in the CIPP and prioritized it as the fourth-most-important item. We identified that the Agency had not completed five CIPP initiatives, including RadNet. We recommended that the Deputy Administrator establish program office accountability for implementing each CIPP initiative as well as milestones, with short- and long-term performance measures for monitoring the implementation of each CIPP initiative. The Agency agreed with the recommendations and established accountability with the Office of Solid Waste and Emergency Response for coordinating all activities related to the CIPP. EPA planned to begin the first phase of deployment for the fixed RadNet monitors in 2006, with completion scheduled in fiscal year 2009. 12-P-0417 4 ------- On January 27, 2009, the OIG issued Report No. 09-P-0087, EPA Plans for Managing Counter Terrorism/Emergency Response Equipment and Protecting Critical Assets Not Fully Implemented. We reported that EPA was behind schedule in implementing RadNet, encountered delays and problems with the administration of the contract for the monitors, and may need to modify installed monitors after completing tests of the design. We recommended that EPA monitor the RadNet contract, develop a schedule for addressing design concerns with the monitor, and oversee the implementation of RadNet against the planned schedule until completed. EPA agreed with the recommendations. An analysis of prior EPA OIG report recommendations is in appendix A. Noteworthy Achievements EPA has made progress in implementing the RadNet CIPP initiative since the OIG's 2006 and 2009 audit reports. As of May 2011, EPA had installed 124 of the 134 monitors it ordered, and it completed a test of the RadNet design. The test determined that a change to the installed monitors was not required. Scope and Methodology We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our review objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. We conducted the audit from May to December 2011. We visited OAM's Headquarters Procurement Operations Division in Washington, DC; OAM's Procurement Operations Division in Research Triangle Park, North Carolina; and NAREL in Montgomery, Alabama. We interviewed the directors for ORIA and NAREL. We interviewed contracting officers (COs) and the RadNet operations manager to follow up on prior report recommendations and agreed-to corrective actions. We conducted interviews with operators for 12 RadNet monitors to determine whether they followed quality control procedures to ensure that data submitted from the monitors were reliable. We selected a random sample of 12 monitors from the 124 installed to identify a population of monitors to analyze for frequency of filter changes and to identify which operators to interview. We reviewed prior audit reports, the EPA 2011-2015 Strategic Plan, OAR's 2011-2012 National Program Manager Guidance, and pertinent laws and regulations. We reviewed information in the Agency's Management Audit Tracking System (MATS), and the CO and contracting officer representative 12-P-0417 5 ------- (COR) files, to determine whether EPA implemented corrective actions to address the recommendations pertaining to RadNet in our 2009 audit report. We reviewed the 2010 and 2012 RadNet Quality Assurance Project Plans (QAPPs) to determine the quality control procedures EPA had to ensure that data submitted from the RadNet are reliable. We reviewed the fiscal years 2010 and 2011 Federal Managers' Financial Integrity Act Annual Assurance Letters for EPA's Office of Homeland Security (within the Office of the Administrator) and OAR to determine whether those two letters identified any weaknesses related to the RadNet program. The letters did not identify such weaknesses. 12-P-0417 6 ------- Chapter 2 EPA Not Managing RadNet as a High-Priority Program Broken RadNet monitors and late filter changes impaired this critical infrastructure asset. On March 11, 2011, at the time of the Japan nuclear incident, 25 of the 124 installed RadNet monitors, or 20 percent, were out of service for an average of 130 days. In addition, 6 of the 12 RadNet monitors we sampled (50 percent) had gone over 8 weeks without a filter change, and 2 had gone unchanged for over 300 days because they were broken. Unless EPA grants an extension, the repair services contract requires a service contractor to fix broken monitors within 14 days of EPA's notification that a monitor is out of service. The EPA 2010 QAPP required operators to change filters on fixed RadNet real-time monitors twice a week. Because EPA did not manage RadNet as a high-priority program, parts shortages and insufficient contract oversight contributed to the extensive delay in fixing broken monitors. Out-of-service monitors and unchanged filters may reduce the availability and quality of critical data needed to assess radioactive threats to public health and the environment. Contract and QAAP Define Repair and Filter Change Time Frames EPA included terms and conditions in the RadNet repair and maintenance services contract to define the period of time for repair. In May 2008, EPA awarded a RadNet repair and maintenance service contract that requires the contractor to fix broken monitors within 14 days of being notified by the COR. EPA may permit an extension of this 14-day period for a specific repair for reasons including unavailability of government-furnished property (GFP), operator unavailability, and physical disruption of the site. EPA acquires the GFP from the contractor that was awarded the propriety spare parts contract. EPA's 2010 QAPP states that RadNet air station operators collect air particulate filters twice a week and mail the filters via the U.S. Postal Service to NAREL for analysis. When elevated levels of radioactivity are anticipated or known to exist, EPA may request RadNet station operators to increase the sampling frequency and use priority shipping. One in Five RadNet Monitors Out of Service an Average of 130 Days At the time of the Japan nuclear incident, 25 of the 124 installed monitors, or 20 percent, were out of service for an average of 130 days (appendix B and figure 2). The 25 out-of-service monitors were located throughout the country, except for the northwest portions of the United States (figure 3). 12-P-0417 7 ------- Figure 2: Length of time RadNet monitors had been broken at time of the Japan nuclear incident 500 400 Average -130 days broken Average - 106 days for repair notification 327 * 294 296 + 300 ~ ~ 200 143 145 145 146 ^ 428 37S..T ~ ~ ~ ~ ~ 100 e-7-72-7-6.. 33 38 39 41 « 47 48 54 ^ « 2( ° 2#6 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 Source: 01G analysis of 25 out-of-service monitors per information provided by EPA. Figure 3: Map of 25 monitors out of service at the time of the Japan nuclear incident Keno Atlantic Ocean Pacific Ocean Source: OIG-developed map using Geographic Information System and Agency-provided information on broken monitors. While the RadNet repair services contract requires the contractor to make the repair within 14 days of being notified by EPA, unless extended, EPA permitted 9 of the 25 monitors to remain out of service for more than 140 days because it did not notify the contractor until parts were available. In addition, two monitors were out of service because no operator was available (table 2). 12-P-0417 8 ------- Table 2: Monitors broken and out of service for more than 140 days Monitor location Date out of service Date back in service Days out of service Harlingen, TX 01/28/10 04/01/11 428 Raleigh, NC 03/29/10 04/08/11 375 Fort Wayne, IN 05/02/10 03/25/11 327 Carlsbad, NM* 06/01/10 03/24/11 296 St. Louis, MO 06/09/10 03/30/11 294 Corpus Christi, TX* 10/01/10 03/21/11 171 Oklahoma City, OK 10/17/10 03/24/11 158 Burlington, VT 11/02/10 03/28/11 146 Fort Smith, AR 11/06/10 03/31/11 145 San Diego, CA 10/26/10 03/20/11 145 St. Paul, MN 11/08/10 03/31/11 143 *No operator available. Source: OIG analysis of 25 out-of-service monitors per information provided by EPA. EPA took an average of 106 days to notify the service contractor about the out-of-service monitors. EPA justified the delayed repairs and late notification as being due to the unavailability of GFP. EPA did not have the required parts to provide to the repair contractor, and would not for an extended period of time. Therefore, EPA did not notify the contractor that these monitors were in need of repair. EPA was not able to furnish the parts required for repair of 22 monitors in a timely manner, had not been able to recruit replacement volunteer operators for 2 monitors, and was able to troubleshoot the problem with the remaining monitor without needing to supply replacement parts. For 8 of the 25 monitors, the EPA COR took 120 to 421 days to notify the repair contractor (table 3). Table 3: Number of days for EPA to notify contractor of needed repairs Monitor location Date out of service Notification date Days to notification Harlingen, TX 01/28/10 03/25/11 421 Raleigh, NC 03/29/10 03/26/11 362 Fort Wayne, IN 05/02/10 03/20/11 322 St. Louis, MO 06/09/10 03/19/11 283 Oklahoma City, OK 10/17/10 03/19/11 153 Burlington, VT 11/02/10 03/20/11 138 Fort Smith, AR 11/06/10 03/24/11 138 San Diego, CA 10/26/10 02/23/11 120 Source: OIG analysis of eight out-of-service monitors per EPA information. 12-P-0417 9 ------- The 25 out-of-service monitors received EPA's priority attention only after the Japan nuclear emergency. The priority attention consisted of cannibalizing monitors under construction for their parts, which were used to repair broken monitors. By April 8, 2011, the service contractor had completed repairs on all monitors (figure 4). Figure 4: Timeline to fix the 25 monitors after the Japan nuclear incident 100% Fercertof 25 Monitors Repaired from the Start of theJapan Nuclear Incident (3/11) Until the Last Monitor was Operational 4/08) . Source: 01G analysis of 25 out-of-service monitors per information provided by EPA. Filters for RadNet Monitors Not Changed Timely Operators are not changing filters for the RadNet monitors two times per week as required by the 2010 QAPP. We sampled 12 out of the 124 installed monitors for which we reviewed the frequency of filter changes for the 12-month period May 1, 2010-April 30, 2011 (table 4). Table 4: Filter changes on random sampling of 12 monitors, May 1, 2010-April 30, 2011 Monitor location No. of filter changes made No. of filter changes not made % not made Longest elapsed time without filter change in days Salt Lake City, UT 61 43 41.35% 129 Riverside, CA 98 6 5.77 N/A Montgomery, AL 103 1 1 N/A Burlington, VT 3 101 97.12 339 Eureka, CA 55 49 47.12 N/A Tulsa, OK 38 66 63.46 152 St. Louis, MO 6 98 94.23 309 Richmond, VA 90 14 13.46 21 Fort Worth, TX 65 39 37.50 21 Omaha, NE 38 66 63.46 147 Houston, TX 74 30 28.85 79 Toledo, OH 97 7 6.73 N/A Average filter changes not made 41.60% Source: OIG analysis of 12 random samples; filter change data provided by EPA. 12-P-0417 10 ------- Our review disclosed that: • Seven of 12 monitors were broken for an extended period and thus did not have filter changes or submit data for 21 to 339 days. • For the 1-year time span, 42 percent of required filter changes did not occur because of broken monitors or volunteer issues. • Monitors located in Burlington, Vermont, and St. Louis, Missouri, had only three and six filter changes for the 1-year period, respectively. Not all operators changed filters twice a week. One operator did not change filters for 309 days because the monitor was broken and unrepaired. Further, because operators are unpaid volunteers from a variety of federal/state/local agencies, universities, and colleges, RadNet filter changes may not be the top priority for all operators. Another operator did not change filters for 339 days because the monitor was broken at one time and the operator had National Guard duty at other times. EPA acknowledged these situations but did not take prompt action. Further, NAREL gave operators permission to wait up to 8 days between filter changes, despite the 2010 QAPP requirement to change the filters twice a week. Some operators requested and NAREL gave permission to change filters once a week. Failure to follow RadNet quality assurance requirements may adversely affect data completeness and potentially impairs RadNet's ability to protect human health. The 2010 QAPP discussed the twice-per-week filter changes in three separates areas (table 5) and addressed specific elements of the planning and implementation of the real-time radiation monitors, including handling of filter samples collected at the air monitoring sites. EPA's 2001 Requirements for Quality Assurance Project Plans, EPA QA/R-5, explained that QAPPs are important because they establish essential components of assuring the credibility and reliability of Agency data and information. The importance of the QAPP is also defined in the May 2000, EPA Quality Manual for Environmental Programs and states: The QAPP is a critical planning document for any environmental data operation since it documents how environmental data operations are planned, implemented, documented, and assessed during the life cycle of a program, project, or task. The ultimate success of an environmental program or project depends on the adequacy and sufficiency of the quality of the environmental data collected and used in decision-making. This may depend significantly on the adequacy of the QAPP and its effective implementation. ThcEPA Quality Manual for Environmental Programs also explains that the QAPP describes the experimental design or data collection design for the project, including as appropriate the types and numbers of samples required, the design of the sampling network, sampling locations and frequencies, sample matrices, measurement parameters of interest, and the rationale for the design. 12-P-0417 11 ------- In response to the draft report, the Agency stated that the frequency of filter changes was not a relevant metric for operational readiness, and stated that the twice weekly filter change referenced in the 2010 QAPP was not intended to be an operational requirement, but rather to provide consistency in throughput at NAREL's analytical laboratory. In March 2012 the Agency revised the QAPP and removed all but one reference to the twice-per-week filter changes, (table 5) Table 5: Information from the 2010 and 2012 QAPPs on filter changes Section QAPP August 6, 2010 QAPP March 8, 2012 8.5.2.2.1 Under routine circumstances, operators send filters, along with field estimates, to NAREL twice weekly from each monitor. Filters are directly counted for gross beta activity at NAREL. 13.1.13, Step 13 Site operator conducts routine twice- weekly filter changes, field analyses, and filter shipment to NAREL Site operator conducts routine filter changes and filter shipment to NAREL. 1.3.1.1.2 Current OMB [Office of Management and Budget] reports for the ERAMS [Environmental Radiation Ambient Monitoring System]/RadNet air monitoring network estimate operator expenditure of time to be 75 person- hours per year. This estimate is based on station operators collecting air filters twice per week. Current OMB reports for the ERAMS/RadNet air monitoring network estimate operator expenditure of time to be 75 person-hours per year. This estimate is based on station operators collecting air filters twice per week. However, less frequent filter changing is common for a variety of logistical reasons and does not affect quality of the data collected by the near-real-time gamma monitoring systems. Source: EPA's 2010 and 2012 QAPPs. Since the 2012 QAPP does not specifically define how often filters should be changed, the Agency needs to define at a minimum how often filter changes are needed to provide consistency in throughput at NAREL's analytical laboratory and also develop a matrix for this. Limited Resources and Competing Priorities Impacted RadNet Readiness Delayed contract awards and staffing issues negatively affected the full implementation of RadNet. The spare parts contract was awarded long after the warranty expired on the first installed monitor, and the RadNet program is run by a single operations manager who has no backup. EPA did not prioritize the award of contracts crucial to support the RadNet program. The first RadNet monitor, installed on April 20, 2006, had a 1-year manufacturer's warranty. However, NAREL did not start the process for the parts contract until May 14, 2008, as part of its fiscal year 2010 Acquisition Forecast Plan, which was about 1 year after the manufacturer's warranty expired on the first installed monitor. NAREL submitted the Procurement Initiation Notice on 12-P-0417 12 ------- July 8, 2009, to the Research Triangle Park Procurement Operations Division. Fourteen months after the notice, on September 21, 2010, EPA awarded the parts contract—over 3 years past the warranty expiration date of the first stationary monitor. The RadNet system is managed by the RadNet Real-Time Air Monitoring operations manager, NAREL. The RadNet operations manager had a variety of responsibilities that competed for prioritization, including but not limited to: • Network and site management for 124 monitors • COR for seven RadNet contracts plus another that is pending (including the three contracts that are the subject of this review) • Quality assurance of RadNet fixed monitor operations Because of this workload, the operations manager had to make judgments about which issues required action on any particular day. Consequently, the operations manager often deferred activities related to monitor operations, especially those with longer-term impact, instead of treating those activities as a priority. For example, the RadNet operations manager explained that he delayed the following long-term tasks because of the immediate demands he had to address: • Maintaining operational status of monitors • Preparing standard operating procedures and quality control documents • Identifying and preparing sites for installation of monitors • Implementing and conducting quality assurance activities The RadNet operations manager has no backup person. The NAREL acting director acknowledged that staffing was a critical area to be resolved, but did not present any specific plans for resolution. Conclusion EPA's RadNet program will remain vulnerable until it is managed with the urgency and priority that the Agency reports it to have to its mission, and that is also reflected in the National Response Framework for Nuclear Radiological Incidents. If RadNet is not managed as a high-priority program, EPA may not have the needed data before, during, and after a critical event such as the Japan nuclear incident. Such data are crucial to determine levels of airborne radioactivity that may negatively affect public health and the environment. 12-P-0417 13 ------- Recommendations We recommend that the Assistant Administrator for Air and Radiation: 1. Establish and enforce written expectations for RadNet operational readiness commensurate with its role in and importance to EPA's mission. Include, at a minimum: a. Percentage of stationary monitors expected to be operational. b. Maximum length of time stationary monitors are permitted to be nonoperational. c. Plan for temporarily backing up broken stationary monitors when operational status is lower than required. d. Availability of monitor operators. 2. Implement metrics for RadNet operational readiness to be reviewed daily by NAREL, and periodically by OAR (at least monthly) and by the Deputy Administrator (as needed), to include, at a minimum: a. Percentage of monitors operational. b. Length of time in nonoperational status. c. Need for backup monitors when operational status is too low. d. Operator availability. 3. Direct that NAREL improve planning and management for RadNet to include, at a minimum: a. Provide for in-stock spare parts to assure operational status established under recommendation 1. b. Implement measures to assure that operators are available. c. How often filter changes are needed to provide consistency in throughput at NAREL's analytical laboratory and implement a metric for these filter changes. Agency Response and OIG Evaluation The Agency concurred with the findings and all but one recommendation, and provided milestone dates for most of the recommendations. The Agency also proposed some revised language, which we incorporated where appropriate in the report. The Agency's full response is in appendix D. The Agency concurred with recommendations l.a. through l.d., and stated that corrective action is expected to be completed by April 1, 2012. We agree with the Agency's corrective actions planned for recommendations l.a. through l.d. 12-P-0417 14 ------- The Agency concurred with recommendation 2.a. through 2.d. and stated that corrective action is expected to be completed by April 1, 2012. We agree with the corrective actions planned for recommendation 2.a through 2.d. The Agency did not agree with our draft report recommendation 2.e. and the need for a performance metric to monitor the frequency of filter changes. The Agency stated the twice weekly filter change referenced in the 2010 QAPP was not intended to be an operational requirement, but rather to provide consistency in throughput at NAREL's analytical laboratory. In March 2012 the Agency revised the QAPP and removed all but one reference to the twice-per-week filter changes. Considering the changes in the 2012 QAPP, we modified and moved recommendation 2.e., to recommendation 3.c., which directs that NAREL improve planning and management for RadNet to include, at a minimum how often filter changes are needed to provide consistency in throughput at NAREL's analytical laboratory and implement a metric for these filter changes, because we believe the Agency needs to define at a minimum, how often filter changes are needed to provide consistency in throughput at NAREL's analytical laboratory. The Agency concurred with recommendation 3.a. and stated that corrective action was completed. We agree with the Agency's corrective action plan for recommendation 3.a. and request that the Agency include the date corrective actions were completed, the inventory of available spare parts, the budget and future funding for spare parts, and the newly awarded repair contract and the statement of work, as soon as this information is available. Also, the Agency concurred with recommendation 3.b. and stated that corrective action is expected to be completed by April 1, 2012. We agree with the Agency's corrective action for 3b. 12-P-0417 15 ------- Chapter 3 EPA Should Improve RadNet Contract Oversight EPA did not fully resolve contracting issues identified in the OIG's January 2009 report. We found that incentives and disincentives for contractors were not included in each of the three RadNet contracts covered by this review, monthly progress reports (MPRs) were not included in terms and conditions of all three contracts, and required contractor performance evaluations were not completed or were late. The OIG's January 2009 report addressed the need to improve contract administration and accountability for the RadNet initiative. EPA concurred with the prior report recommendations and established corrective action plans to monitor and improve contractor performance and oversight issues. The report contained five recommendations, of which EPA has listed four in MATS as completed as of August 2011. However, OIG analysis demonstrated that EPA only completed one of the recommendations (appendix A). As a result, contractor performance issues remained because EPA did not implement prior report recommendations to hold contractors accountable. Until EPA corrects the shortfall in contract oversight, the ability of the RadNet to protect human health and the environment is potentially impaired. Federal and EPA Guidance Define Contract Management Requirements Federal and EPA regulation and guidance are available to assist EPA in managing contracts and holding contractors accountable: • Federal Acquisition Regulation (FAR) 16.401—Incentive Contracts: FAR 16.401 discusses the need to provide motivation (incentives) for a contractor to do well. It states that incentive contracts are designed to obtain specific acquisition objectives by including appropriate incentive arrangements designed to (1) motivate contractor efforts that might not otherwise be emphasized, and (2) discourage contractor inefficiency and waste. • FAR 16.403—Incentive Contracts: FAR 16.403 states a fixed-price incentive contract is appropriate when the nature of the supplies or services being acquired and other circumstances of the acquisition are such that the contractor's assumption of a degree of cost responsibility will provide a positive profit incentive for effective cost control and performance. 12-P-0417 16 ------- • FAR 42.11—Surveillance Requirements: FAR 42.1104 states that the extent of production surveillance is determined by the contract administration office on the basis of degree of importance to the government, contract requirements for reporting production progress and performance, the contract performance schedule, the contractor's history of contract performance, and the contractor's experience with the contract supplies or services. • FAR 42.15—Contractor Performance Information: FAR 42.1502(c) states that, for each contract and task/delivery order in excess of the simplified acquisition threshold of $150,000, the CO shall annually prepare an evaluation of the contractor's performance. • Environmental Protection Agency Acquisition Regulations (EPAAR) Deviation 1542.15 —Contractor Performance Information:3 Evaluation reports shall be submitted to the Defense Department's Past Performance Information Retrieval System (PPIRS) through the Web- based Contractor Performance Assessment Reporting System (CPARS), which has connectivity with PPIRS. CPARS replaced the previous reporting system, the National Institutes of Health (NIH) Contractor Performance System (CPS), on May 15, 2010. An evaluation covers each 12-month period after the effective date of the contract or order. EPA's Interim Policy Notice 10-03 required completion of evaluations within 90 business days from the date the CO initiates the evaluation. Incentives/Disincentives Not Included in All RadNet Contracts The fixed air monitoring stations contract was the only one of the three in our review that contained an incentive. It was also the only one that included a disincentive; however, we consider that disincentive to be ineffective (table 6). Table 6: Incentives and disincentives in the three RadNet contracts under review Contract Incentive Disincentive Fixed air monitoring stations Yes Yes, ineffective Repair service4 No No Spare parts No No Source: OIG analysis. 3 EPAAR 1542.15, Contractor Performance Information, superseded and rescinded OAM Interim Policy Notice 10-03, effective October 3, 2011. 4 In the Agency response to the draft report, the Agency stated that 3 days prior to release of our draft report to OAR and OAM, EP-D-I2-003 was awarded on December 12, 2011, for RadNet Air Monitor Maintenance. This contract replaced EP-D-08-068 and does include disincentives for subpar performance as follows: a 5 percent reduction on the invoiced labor amount for eveiy 1 percent slippage from the 95 percent performance level. 12-P-0417 17 ------- The OIG's January 2009 report recommended that the Agency maintain current incentives in the new RadNet contracts and seek opportunities to expand these, and include disincentives in future contracts of this type; i.e., when appropriate, obtain reasonable equitable adjustments to the contract as a remedy for subpar contractor performance. The Agency reported this recommendation as completed in MATS (appendix A, recommendation 2-1). Fixed Air Monitoring Stations Contract Did Not Include Effective Disincentive Of the three RadNet contracts, only the contract for fixed air monitoring stations contained incentives in the contract's Quality Assurance Surveillance Plan, as encouraged by FAR 16.401 and 16.403. Under the plan, the contractor had the potential to earn an additional $500 to $1,000 for making warranty repairs earlier than the 30 days required by the contract. The performance standards in the Quality Assurance Surveillance Plan under the contract for fixed air monitoring stations listed a positive past performance evaluation as an incentive/disincentive. However, positive past performance evaluation is not a true incentive/disincentive, because performance evaluation is a FAR 42.15 requirement. Even though the OIG raised the issue of disincentives in its January 2009 report, EPA did not add to the contract disincentives if the contractor made warranty repairs after the 30-day requirement. EPA did not effectively use disincentives. Under Delivery Orders 1 and 2, EPA had to extend the contractor period of performance to allow the contractor more time to deliver the required monitors. As a result, EPA received compensation valued at approximately $41,830 (appendix C). We do not consider this to be an effective disincentive because there were no enforceable contract requirements for EPA to receive compensation for untimely delivery. Planned Disincentives Not Included in Spare Parts Contract EPA did not include disincentives in the September 2010 RadNet spare parts contract. NAREL submitted a statement of work (SOW) for the spare parts contract to the OAM Procurement Operations Division in Research Triangle Park, which included discounts for untimely delivery. Specifically, the SOW included the following: • Items shall be delivered within 90 days of order. • Any item delivered after 90 days of order shall be invoiced at the following discount: > Delivery of any item between 91 and 120 days of order shall be invoiced with a discount of 2 percent. 12-P-0417 18 ------- > Delivery of any item after 120 days of order shall be invoiced with a discount of one additional percentage (1 percent) per additional 30-day delay. However, none of the discounts outlined in the SOW were included in the awarded spare parts contract. The CO stated that the contract included terms for volume discounts, but the delivery discounts in the SOW were not included because the Procurement Operations Division decided to use a clause in the contract to allow the NAREL COR to determine delivery terms. However, the COR did not specify delivery terms to the contractor. On March 9, 2011, EPA modified the contract to approve the 2011 price list. This modification included time frames for delivery, but still did not include any of the initial discounts outlined in the SOW for untimely delivery. Without using the types of disincentives outlined in the initial SOW, EPA has no readily quantifiable measure to hold the contractor accountable for timely completion of contract activities. Installation of RadNet Monitors Behind Schedule Contract oversight and contractor performance issues continue to hinder the delivery of RadNet monitors. As of November 30, 2011, EPA was 2 years and 5 months behind its initial schedule to install monitors established in the fixed air monitoring stations contract terms and conditions (table 7). Table 7: Schedule of fixed air monitoring stations ordered and received Delivery Order Date awarded Number ordered Number delivered Required start of delivery Delivery order end date EPA extended date for delivery Date of last shipment Late delivery timing 1 10/01/2007 51 51 01/31/2008 12/31/2008 04/30/2009 05/19/2009 4.6 months 2 06/10/2008 32 22 01/01/2009 07/31/2009 11/30/2011 07/08/2010 2 years, 5 months Source: Delivery Orders 1 and 2, modifications, and OIG analysis. Note: This contract was for 83 monitors, split between two delivery orders. The other 51 monitors were covered under an earlier contract. Per contract terms, the delivery and the ship dates are the same. Delivery orders under the contract required the delivery of five monitoring stations per month beginning in January 2008. The OIG's January 2009 report recommended that the Agency track the installation of the RadNet system against the planned schedule in the CIPP until completed. The Agency has not reported this recommendation as completed in MATS; OAR recorded in MATS that it expected to complete monitor installation in spring 2011 (appendix A, recommendation 2-5). However, because of challenges brought on by the Japan nuclear incident, OAR now expects to complete installation of the remaining 10 monitors by June 2012. 12-P-0417 19 ------- EPA extended the period of performance for the delivery of the monitors to give the contractor additional time to deliver the required monitors. EPA extended Delivery Order 1 twice and Delivery Order 2 four times, with a final extension date of November 30, 2011, for Delivery Order 2. EPA (figure 5, left) was responsible for installation delays, and the contractor (figure 5, right) was responsible for delivery delays. Figure 5: Delays of delivery and installation of RadNet monitors S 120 17 25 33 41 49 57 Shipped Monitors HI - #73 Contracted Shipped Q - <-> Q - Date Contracted & Shioped Source: OIG analysis of Delivery Orders 1 and 2, and EPA-provided data on shipment monitors. of For EPA's part, NAREL was required to have five locations available per month for the contractor to deliver and install RadNet monitors at the required rate of five per month. However, the RadNet operations manager stated that he was unable to complete all the requirements at a rate of five locations per month, which delayed the delivery and installation of some of the monitors. Delivery and installation of RadNet monitoring stations by the contractor to a site requires NAREL to have identified sites and operators within cities designated by EPA, and to arrange for site preparation. To select operators, NAREL works with the EPA regions and departments of health in the states to find a volunteer to operate the monitor (i.e., change filters and submit data). Operators and state offices are not compensated. Site preparation may include installing electrical outlets, constructing mounting platforms, or erecting security fencing. For the contractor's part, a subcontractor in Italy was unable to timely submit two proprietary components—local processor units and detectors. Further, in March 2011, EPA directed that components from the final 10 monitors under construction be used to fill a delivery order under the separate spare parts contract to provide parts needed to repair broken monitors that were already installed. This 12-P-0417 20 ------- diversion of components away from construction of the final 10 monitors in Delivery Order 2 provided the spare parts contractor with enough equipment to repair the 25 monitors that were not working at the time of the Japan nuclear incident. EPA is addressing the difficulty of acquiring proprietary spare parts with a foreign subcontractor by exploring domestic options. The December 2011 SOW for the newly awarded RadNet service contract5 contains language to address the proprietary issue. For example, task 4 states: The contractor shall identify components that have or may develop limited availability or serviceability, and the contractor shall identify more readily available, improved or more serviceable components that may be potentially used to replace original components. Monthly Progress Reports Not Effectively Used to Manage RadNet Contracts EPA is not fully monitoring contractor performance through MPRs as required by FAR 42.11. Two of three RadNet contracts required MPRs; however, EPA required only one contractor to actually submit them (table 8). Table 8: MPR requirements in the three RadNet contracts under review Contract MPR status Fixed air monitoring stations Required Spare parts Not required Repair service Required, but not enforced Source: OIG analysis. The OIG's January 2009 report recommended that the Agency use MPRs to monitor actual contractor performance against stated goals. The Agency reported this recommendation as completed in MATS (appendix A, recommendation 2-2). The fixed air monitoring stations contract required the contractor to provide MPRs. NAREL received them as required, and they contained general information on contractor performance. The spare parts contract did not include a requirement for MPRs; therefore, the contractor was not submitting them. Given this contractor's history of untimely delivery under the fixed air monitoring stations contract, EPA should have been monitoring the contractor's performance through MPRs. 5 In the Agency response to the draft report, the Agency stated that the contract was awarded in December 2011 12-P-0417 21 ------- The repair services contract included the following MPR requirement: "within the seven days of the end of each month, the Contractor shall provide a written report to the COR describing work performed during the month." However, the Agency did not require MPRs from the contractor. The COR stated that he required a repair report after each repair made, but did not require MPRs. As a result, EPA is not getting updated contractor activity as required, and documented contractor performance information is not available for use in contractor performance evaluations. Contractor Performance Evaluations Not Timely or Completed As of October 12,2011, EPA had not timely completed four of five required contractor performance evaluations for the RadNet contractors. The OIG's January 2009 report recommended that the CO and COR formally evaluate the fixed air monitoring station contractor's performance on an annual basis and enter past performance information into the NIH CPS under the expired and current contract. The Agency reported this recommendation as completed in MATS (appendix A, recommendation 2-3). To ensure compliance with FAR 42.15, COs were required to include EPAAR Clause 1552.209-76 in their contracts, which requires annual contractor performance evaluations. Effective May 15, 2010, EPAAR Deviation 1542.15 replaced EPAAR 1509.170, to require the entry of the contractor evaluations into PPIRS instead of NIH CPS. Contractor performance evaluations in PPIRS are available to all federal contracting offices nationwide. Contractor past performance is one of many factors to consider when assessing whether contractors are likely to be successful in controlling contract costs and meeting contract expectations. When EPA documents evaluations in an untimely manner or not at all, selection teams within EPA and at other federal agencies do not have current contractor evaluation information to consider when making new contract awards. As of October 12, 2011, EPA should have performed five contractor performance evaluations: three for the fixed air monitoring stations contract and two for the repair services contract.6 However, only one for the fixed air monitoring stations contract was completed (table 9). 6 The spare parts contractor performance evaluation was not due until after our review period ended. The contract was awarded in September 2010; the evaluation period covers 1 year and is due 90 days after the evaluation period ends. 12-P-0417 22 ------- Table 9: Timeliness of contractor performance evaluations Contract Required period Date due Completed Period covered Days past due as of 10/12/2011 EP-W-07-076 Fixed Air Monitoring Stations 09/28/07-09/27/08 02/12/09 01/30/09 09/27/07-12/30/08® N/A 09/28/08-09/27/09 02/11/10 No N/A 420 09/28/09-09/27/10 02/13/11 No N/A 168 EP-D-08-068 Repair Services 05/12/08-05/11/09 09/24/09 No N/A 513 05/12/08-05/11/10 09/24/10 No N/A 262 Source: OIG analysis of contractor performance evaluations as of October 12, 2011. a The initial evaluation improperly covered 15 months; the regulation requires the evaluation to cover a maximum period of 12 months. The CO for the fixed air monitoring stations contract stated that conducting the contractor evaluation was not a priority. Further, during the transition from the NIH CPS system to CPARS/PPIRS, the COs had only limited access to CPARS, yet were required to enter more data than was previously required. EPA conducted none of the required evaluations for the repair services contract, and the requirements for evaluations were not included in the terms and conditions of the contract. The CO stated that he would conduct the evaluation on close-out of the contract. In May 2009, the COR reminded the CO to complete the required performance evaluation. However, as of October 12, 2011, none of the required performance evaluations had been conducted. Conclusion OAM and OAR did not adequately oversee the three RadNet contracts we reviewed and did not fully use contract requirements, including using contract incentives and disincentives, MPRs, and annual performance evaluations, to hold contractors accountable. As a result, contract issues raised in our January 2009 report continue to exist because EPA believed it could oversee and hold contractors accountable without them. EPA should implement recommended actions from this report to hold RadNet contractors accountable. Doing so will help ensure that EPA's network of monitors is fully installed and operational so it can meet requirements established in the National Response Framework for Nuclear Radiological Incidents. 12-P-0417 23 ------- Recommendations We recommend that the Assistant Administrator for Air and Radiation, in conjunction with the Assistant Administrator for Administration and Resources Management: 4. Require follow-on RadNet contracts to include incentives/disincentives and a requirement for MPRs. 5. Require the CO and COR to formally evaluate RadNet contractors' performance on an annual basis and enter information into PPIRS through CPARS. 6. Determine whether domestic contract options are available for crucial repair parts that are identified as only being available from a foreign subcontractor. 7. Review the information in MATS for the prior audit and ensure it is accurate and current. We recommend that the Assistant Administrator for Air and Radiation: 8. Track the installation of the RadNet monitors against the revised schedule and use the contract requirements in recommendations 4 and 5 to hold the contractor accountable. Agency Response and OIG Evaluation The Agency generally concurred with the findings and recommendations, and provided milestone dates for most of the proposed corrective actions. The Agency also proposed some revised language, which we incorporated where appropriate in the report. The Agency's full response is in appendix D. The Agency partially concurred with recommendation 4 to require that RadNet contracts include incentives/disincentives and require MPRs for follow-on contracts, but not for existing contracts. We agree with the Agency corrective action for recommendation 4 and have revised the recommendation. We request that the Agency provide the awarded contract and statement of work for contract EP-D-12-003 and the follow-on contract EP-D-10-0085 as soon as the information becomes a available. The Agency partially agreed with recommendation 5 to require the CO and COR to formally evaluate RadNet contractors' performance on an annual basis. The Agency stated that it will track and report timely completion of contractor performance evaluations under a Balanced Scorecard Internal Business Performance Measure and that new and/or ongoing contracts will receive priority 12-P-0417 24 ------- for completing past performance reporting over expired contracts. The Agency further stated that a target of not less than 90 percent of past performance evaluations is to be completed in CPARS and contractor performance evaluations will be brought up to date as applicable. We continue to recommend that EPA formally evaluate RadNet contractors' performance, even if the contract has expired, and enter information into PPIRS through CPARS. FAR does not make exceptions for contractor evaluation based on whether contracts are active or expired. We request that the Agency include a date for doing so in the 90-day response to the final report. The Agency agreed with recommendation 6 and stated that it is conducting additional market research in accordance with FAR Part 10 to identify potential domestic sources prior to the re-compete of the spare parts contract. This contract is anticipated to be awarded prior to the end of fiscal year 2012. We agree with the Agency corrective action for recommendation 6. The Agency agreed with recommendation 7 and provided the status of recommendations from OIG Report No. 09-P-0087. We agree with the Agency corrective action for recommendation 7 and request that the Agency include information on the finalized February and April 2012 performance evaluations under contract EP-W-07-076 in the 90-day response to the final report. The Agency concurred with recommendation 8 and stated that the COR is working with the CO to enforce the terms and conditions of the contract including receipt of consideration for late deliveries. We agree with the Agency corrective plan for recommendation 8. However, since the expected completion date of installation of monitors is now June 2012, we request that the Agency include information in the 90-day response to the final report on any consideration the Agency received for the additional delayed delivery of the monitors. 12-P-0417 25 ------- Status of Recommendations and Potential Monetary Benefits RECOMMENDATIONS POTENTIAL MONETARY BENEFIT (in $000s) Planned Rec. Page Completion Claimed Agreed-To No. No. Subject Status1 Action Official Date Amount Amount 14 Establish and enforce written expectations for RadNet operational readiness commensurate with its role and importance to EPA's mission. Include, at a minimum: a. Percentage of stationary monitors expected to be operational. b. Maximum length of time stationary monitors are permitted to be nonoperational. c. Plan for temporarily backing up broken stationary monitors when operational status is lower than required. d. Availability of monitor operators. 14 Implement metrics for RadNet operational readiness to be reviewed daily by NAREL, and periodically by OAR (at least monthly) and by the Deputy Administrator (as needed), to include, at a minimum: a. Percentage of monitors operational. b. Length of time in nonoperational status. c. Need for backup monitors when operational status is too low. d. Operator availability. 14 Direct that NAREL improve planning and management for RadNet to include, at a minimum: a. Provide for in-stock spare parts to assure operational status established under recommendation 1. b. Implement measures to assure that operators are available. c. Mow often filter changes are needed to provide consistency in throughput at NAREL's analytical laboratory and implement a metric for these filter changes. 24 Require follow-on RadNet contracts to include incentives/disincentives and a requirement for MPRs. Assistant Administrator for Air and Radiation Assistant Administrator for Air and Radiation Assistant Administrator for Air and Radiation Assistant Administrator for Air and Radiation, in conjunction with the Assistant Administrator for Administration and Resources Management 04/01/12 04/01/12 04/09/12 04/01/12 9/30/12 12-P-0417 26 ------- RECOMMENDATIONS POTENTIAL MONETARY BENEFIT (in $000s) Planned Rec. Page Completion Claimed Agreed-To No. No. Subject Status1 Action Official Date Amount Amount 24 Require the CO and COR to formally evaluate RadNet contractors' performance on an annual basis and enter information into PPIRS through CPARS. 24 Determine whether domestic contract options are available for crucial repair parts that are identified as only being available from a foreign subcontractor. 24 Review the information in MATS for the prior audit and ensure it is accurate and current. 24 Track the installation of the RadNet monitors against the revised schedule and use contract requirements in recommendations 4 and 5 to hold the contractor accountable. Assistant Administrator for Air and Radiation, in conjunction with the Assistant Administrator for Administration and Resources Management Assistant Administrator 09/30/12 for Air and Radiation, in conjunction with the Assistant Administrator for Administration and Resources Management Assistant Administrator 06/30/12 for Air and Radiation, in conjunction with the Assistant Administrator for Administration and Resources Management Assistant Administrator 06/30/12 for Air and Radiation 1 O = recommendation is open with agreed-to corrective actions pending C = recommendation is closed with all agreed-to actions completed U = recommendation is unresolved with resolution efforts in progress 12-P-0417 27 ------- Appendix A Analysis of Prior EPA OIG Report Recommendations Information from MATS OIG # OIG recommendations Completed Actions taken analysis 2-1 Maintain current incentives in the new RadNet contract and seek opportunities to expand these and include disincentives in future contracts of this type. When appropriate, obtain reasonable equitable adjustments to the contract as a remedy for subpar contractor performance. 2009-01-30 The COR is evaluating contractor performance on a monthly basis for potential subpar performance by methods described in 2-2. If subpar performance is identified, the CO will seek reasonable equitable adjustment to the contract as a remedy. Not complete 2-2 Use the monthly progress reports to monitor actual contractor performance against stated goals. 2009-01-30 The COR is conducting weekly scheduled telephone meetings and will discuss any discrepancies between actual performance and stated goals with the contractor. These discrepancies will be reported to the CO. The COR is receiving monthly progress reports with clear description of contracted and actual delivery dates. Not complete 2-3 Require the CO and COR to formally evaluate the contractor's performance on an annual basis and enter past performance information into the National Institutes of Health's (NIH's) Contractor Performance System under the expired and current contract. 2008-04-24 2009-01-30 We completed the contractor performance evaluation for expired contract and delivery order and submitted it to the CO on April 24, 2008, for entry into NIH's Contractor Performance System. Contractor performance evaluation for the current contract and its first delivery order is currently in progress and we will submit it to the CO for entry into NIH's system by January 30, 2009 (30 days after delivery order due date) Not complete 2-4 Establish a plan, with milestone dates, for completing the SAB recommended testing and, if needed, develop and implement a plan for making design improvements. 2009-07-30 Issue report with evaluation of test data and recommendations on design changes. The report did not recommend any design changes. Complete 12-P-0417 28 ------- # OIG recommendations Information from MATS OIG analysis Completed Actions taken 2-5 Monitor the upgrade of the RadNet system against the planned schedule in the CIPP until completed. Not completed OAR originally expected to complete the installation of monitors in the spring 2011. However, because of challenges brought on by the Japan incident, OAR now expects to complete installation of the monitors by the end of 2011. The network will include a total of 134 monitors. The reduction in the total number of monitors purchased is due to budget reductions as well as meeting our long-term performance target of monitors within the 100 most populous cities. In addition, with 134 monitors we also meet our long- term goal of population coverage within 25 miles of a monitor (55%). Final corrective action for this review will be completed by September 30, 2012. Not completed Source: EPA-re ported information from MATS. 12-P-0417 29 ------- Appendix B Analysis of 25 Out-of-Service Monitors as of March 11, 2011—Date of Japan Nuclear Incident Monitor No. Monitor location Date out of service Notification date3 Days to notification Date back in service Days out of service EPA was unable to provide government-furnished parts 1 Harlingen, TX 01/28/10 03/25/11 421 04/01/11 428 2 Raleigh, NC 03/29/10 03/26/11 362 04/08/11 375 3 Fort Wayne, IN 05/02/10 03/20/11 322 03/25/11 327 4 St. Louis, MO 06/09/10 03/19/11 283 03/30/11 294 5 Oklahoma City, OK 10/17/10 03/19/11 153 03/24/11 158 6 Burlington, VT 11/02/10 03/20/11 138 03/28/11 146 7 Fort Smith, AR 11/06/10 03/24/11 138 03/31/11 145 8 San Diego, OA 10/26/10 02/23/11 120 03/20/11 145 9 St. Paul, MN 11/08/10 01/12/11 65 03/31/11 143 10 Philadelphia, PA 12/28/10 01/12/11 15 03/14/11 76 11 Hartford, CT 12/29/10 01/13/11 15 03/11/11 72 12 Lubbock, TX 01/22/11 02/23/11 32 03/30/11 67 13 Syracuse, NY 01/23/11 03/20/11 56 03/18/11 54 14 Chicago, IL 02/02/11 03/20/11 46 03/22/11 48 15 Milwaukee, Wl 02/05/11 03/20/11 43 03/24/11 47 16 El Paso, TX 02/09/11 02/23/11 14 03/28/11 47 17 Phoenix, AZ 02/11/11 02/23/11 12 03/24/11 41 18 Buffalo, NY 02/19/11 03/25/11 34 03/30/11 39 19 Fontana, CA 02/11/11 02/23/11 12 03/21/11 38 20 Reno, NV 02/19/11 03/19/11 28 03/24/11 33 21 Memphis, TN 03/05/11 03/25/11 20 03/31/11 26 22 Laredo, TX 03/04/11 N/Ab - 03/15/11 11 No operator 23 Carlsbad, NM 06/01/10 N/A N/A 03/24/11 296 24 Corpus Christi, TX 10/01/10 N/A N/A 03/21/11 171 A definite "out-of-service" date not easily defined; monitor running erratically/intermittently 25 Yuma, AZ 02/25/11 02/23/11c -2 03/17/11 20 Average days to notify service contractor: 106 Average days out of service: 130 Source: OIG analysis of 25 out-of-service monitors per information provided by EPA. a Date that EPA notified the contractor that the monitor was out of service. b No notification went out. On March 4, 2011, the operations manager and the operator began troubleshooting the problem, which resulted in the return to service on March 15, 2011. c EPA notified the contractor by phone 2 days earlier than the officially recorded out-of-service date. 12-P-0417 30 ------- Appendix C Extensions to the Fixed Air Monitoring Stations Contract and EPA Compensation Received Modifi- cation No. Extended date for delivery of monitor Purpose of extension Value of compen- sation Actual compensation received Delivery Order 1—period of performance: 10/01/07 to 12/31/08 $3,011,902 paid to contractor as of 09/01/2011 2 02/27/2009 Delivery arid installation of monitors $14,200 Upgrade to front end software 3 04/30/2009 Delivery and installation of monitors N/A Warranties for nine monitors extended from 12 months to18 months 4 04/30/2009 Extend the date for completion of warranty repairs for five defective monitors $14,416 Provide two MAB units and a 12-hour "test run" for each new monitor Delivery Order 2—period of performance: 01/01/09 to 07/31/09 $1,457,804 paid to contractor as of 09/01/2011 1 10/30/2009 Delivery delays from a subcontractor of LPUs and detectors $13,187 Repair/replacement of the gamma detector of the monitor located at NAREL 3 10/30/2010 EPA unable to provide all delivery addresses to the contractor N/A N/A 4 04/30/2011 Excusable delays clause— fire destroyed a subcontractor facility N/A N/A 5 11/30/2011 Provide time for the contractor to reconstruct the final 10 monitors N/A N/A Total EPA compensation received $41,803 Source: Modifications to Delivery Orders 1 and 2 of the fixed air monitoring stations contract and OIG analysis. 12-P-0417 31 ------- Appendix D Agency Response to Draft Report ($§£) pro** UNITED STATES ENVIRONMENTAL PROTECTION AGENCY WASHINGTON, D.C. 20460 February 3, 2012 MEMORANDUM SUBJECT: Response to OIG Draft Project Report No. OA-FY11 - 0184 FROM: Gina McCarthy, Assistant Administrator Office of Air and Radiation Craig Hooks, Assistant Administrator Office of Administration and Resource Management TO: Melissa M. Heist, Assistant Inspector General for Audit Office of the Inspector General This memorandum is in response to the Office of Inspector General's (OIG's) request for comments on the draft project report dated December 15, 2011: Weaknesses in EPA's Management of the Radiation Network System Demand Attention. As requested by the OIG, the Office of Air and Radiation (OAR) and the Office of Administration and Resource Management (OARM) and transmit the Agency's consolidated response to this report as an attachment to this memorandum. While we agree with most of the proposed recommendations, we have identified some revisions in the draft report, as well as some suggested language to assist the OIG in their final report. Per your request, we are also providing planned completion dates for all agreed-to recommendations Please feel free contact us directly, or your staff may contact Jonathan Edwards, at (202) 343- 9437, if you have any questions. Attachment cc: Mike Davis 12-P-0417 32 ------- RESPONSE TO OIG DRAFT REPORT (Project No. OA-FY11-0184) This response is organized into two main sections. The first section combines all of the draft report's recommendations and related responses. The second section provides the EPA's other comments which focus largely on contract matters, sampling frequency and filter changes and their relationship to data quality, and the RadNet's Quality Assurance Manual and Quality Assurance Project Plan. In many cases, the EPA comments are accompanied by suggested revisions to clarify perceived misunderstandings or to correct inaccuracies. The EPA is particularly concerned about the statements concerning "relaxed quality controls" since the EPA contends that this is inaccurate, as described in detail in the appropriate places in this response. For convenience, red text is used to indicate some suggested revisions. Responses to OIG Recommendations Recommendation la: Establish and enforce written expectations for RadNet operational readiness commensurate with its role in and importance to EPA's mission. Include, at a minimum: Percentage of stationary monitors expected to be operational. Response: The EPA concurs and has been reevaluating its current operational goal, and will finalize this effort by April 1, 2012. Recommendation lb: Establish and enforce written expectations for RadNet operational readiness commensurate with its role in and importance to EPA's mission. Include, at a minimum: Maximum length of time stationary monitors are permitted to be nonoperational. Response: EPA concurs and has established a maximum length of time for RadNet fixed monitors to be nonoperational before reporting to ORIA and OAR management, along with a process for evaluating and reporting the circumstances associated with the nonoperational status. The reporting requirement will begin April 1, 2012. Recommendation lc: Establish and enforce written expectations for RadNet operational readiness commensurate with its role in and importance to EPA 's mission. Include, at a minimum: Plan for temporarily backing up broken stationary monitors when operational status is lower than required. Response: EPA concurs and has evaluated various options for backing up broken stationary monitors, and will complete the written documentation and plan by April 1, 2012. Recommendation Id: Establish and enforce written expectations for RadNet operational readiness commensurate with its role in and importance to EPA's mission. Include, at a minimum: Availability of monitor operators. Response: EPA concurs with the importance of monitor operator availability and will finalize written plans for maximizing operator availability by April 1, 2012. However, EPA cannot assign volunteers or enforce expectations upon them. Instead, EPA seeks 12-P-0417 33 ------- volunteers, without compensation, for their time and effort. RadNet personnel work closely with their partners, particularly the EPA regions, to do their best in recruiting volunteer operators. When a suitable volunteer operator is identified, EPA also requests a backup volunteer who works with the primary operator to maximize operator availability. Most locations have a backup operator identified and fully trained in all aspects of monitor operations. Once EPA finds an operator, the RadNet volunteer coordinator provides information to them routinely and is in frequent contact by phone. EPA also provides recognition, such as letters of appreciation to their supervisors, for their service in an effort to maintain a good relationship with our volunteers. The response to recommendation 2d contains provisions for reporting to senior EPA management when operators are unavailable for an extended period of time. Recommendation 2a: Implement metrics for RadNet operational readiness to be reviewed daily byNAREL, and periodically by OAR (at least monthly) and by the Deputy Administrator (at least quarterly), to include, at a minimum: Percentage of monitors operational. Response: EPA concurs with the importance of maintaining operational readiness metrics. NAREL will continue to monitor, measure and review RadNet operational readiness every business day. The percentage of operating monitors will be included in written reports, developed using the following process: • weekly reports on metrics for ORIA; • monthly status summary reports for review by ORIA and OAR management; and • OAR management will advise the Deputy Administrator, when deemed appropriate, This reporting process will be established and begin April 1, 2012. Recommendation 2b: Implement metrics for RadNet operational readiness to be reviewed daily byNAREL, and periodically by OAR (at least monthly) and by the Deputy Administrator (at least quarterly), to include, at a minimum: Length of time in nonoperational status. Response: EPA concurs with the importance of maintaining operational readiness metrics. The reports described in response to recommendation 2a will include a list of the operational status of all monitors, along with the repair status and anticipated date for non-operational monitors to return to service. This reporting process will be established and begin April 1, 2012. Recommendation 2c: Implement metrics for RadNet operational readiness to be reviewed daily byNAREL, and periodically by OAR (at least monthly) and by the Deputy Administrator (at least quarterly), to include, at a minimum: Need for backup monitors when operational status is too low. Response: EPA concurs with the importance of maintaining operational readiness. This expectation is being met through the repair of out-of-service monitors. The reports described in response to recommendation 2a will be informed by the evaluation of options for backing up broken monitors, as described in response to recommendation lc. This reporting process will be established and begin April 1, 2012. 12-P-0417 34 ------- Recommendation 2d: Implement metrics for RadNet operational readiness to be reviewed daily byNAREL, and periodically by OAR (at least monthly) and by the Deputy Administrator (at least quarterly), to include, at a minimum: Operator availability. Response: EPA concurs with the importance of maintaining operational readiness metrics. This information will be included in the monthly reports described in 2a. This information will also be forwarded to the appropriate EPA Regional Radiation Managers. This process will be established and begin April 1, 2012. Recommendation 2e: Implement metrics for RadNet operational readiness to be reviewed daily byNAREL, and periodically by OAR (at least monthly) and by the Deputy Administrator (at least quarterly), to include, at a minimum: Frequency of filter changes per the QAM and QAPP. Response: EPA concurs with the importance of operational readiness metrics. However, the frequency of filter changes is not a relevant metric for operational readiness. For clarification, the Quality Assurance Manual (QAM) described in the draft OIG report does not apply to the RadNet fixed real-time monitors; it explicitly excludes them, and notes that the routine operations Quality Assurance Project Plan (QAPP) for RadNet real- time monitors is the applicable quality document. Also for clarification, the twice weekly filter change references in this QAPP are intended to provide consistency in throughput at NAREL's analytical laboratory, not as an operational requirement. The QAPP for RadNet real-time monitors has been edited to clarify this intent. Recommendation 3a: Direct that NAREL improve planning and management for RadNet to include, at a minimum: Provide for in-stock spare parts to assure operational status established under recommendation 1. Response: EPA concurs and the corrective action has been completed. The spare parts contract is now in place, there is an inventory of spare parts, and funding is budgeted for additional inventory of proprietary spare parts. Funding for spare parts for future years is also included in NAREL's projected long-term RadNet budget. This is based upon repair rates to date coupled with aging of the monitors. Additionally, NAREL is pursuing the required funding to have the repair contractor investigate lower cost/higher availability spare parts that can replace the proprietary spare parts. Recommendation 3b: Direct that NAREL improve planning and management for RadNet to include, at a minimum: Implement measures to assure that operators are available. Response: EPA concurs with the importance of maximizing operator availability and will explore measures by April 1, 2012. However, EPA cannot assign volunteer operators or enforce availability of volunteer operators. Instead, EPA seeks volunteers, without compensation, for their time and effort. RadNet personnel work closely with their partners, particularly the EPA regions, to recruit volunteer operators. When a suitable volunteer operator is identified, EPA also requests a backup volunteer who works with the primary operator to maximize operator availability. Most locations have a backup 12-P-0417 35 ------- operator identified and fully trained in all aspects of monitor operations. Once EPA finds an operator, the RadNet volunteer coordinator provides information to them routinely and is in frequent contact by phone. EPA also provides recognition, such as letters of appreciation to their supervisors, for their service in an effort to maintain a good relationship with our volunteers. The response to recommendation 2d contains provisions for reporting to senior EPA management when operators are unavailable for an extended period of time. Recommendation 4: Modify existing and require follow-on RadNet contracts to include incentives/disincentives and a requirement for Monthly Performance Reviews (MPRs). Response: EPA concurs with the draft report findings pertaining to EP-W-07-076 for Fixed Air Monitoring Stations. As this contract ends on March 31, 2012 (Delivery Order 3), there is no meaningful performance period remaining against which to apply the recommendation under the existing contract. EPA also concurs with the draft report findings pertaining to EP-D-08-068 for repair and maintenance services. However this contract has expired, and follow-on contract EP-D- 12-003, awarded on December 12, 2011, for RadNet Air Maintenance, includes detailed performance metrics and provides for a deduction in the invoiced amount for failure to meet those targets. Contract EP-D-12-003 also contains detailed invoice reporting requirements which duplicate the information included in an MPR, thereby satisfying the recommendation. With respect to contract EP-D-10-0085, again the EPA concurs with draft report findings. The follow-on contract is currently in the planning stages for award prior to the end of FY 2012. The resultant contract will include appropriate incentives, quality control requirements, and reporting requirements consistent with this recommendation. Recommendation 5: Require the CO and COR to formally evaluate RadNet contractors' performance on an annual basis and enter information intoPPIRS through CPARS. Response: The EPA will track and report timely completion of contractor performance evaluations under a Balanced Scorecard Internal Business Performance Measure. This will require 100% of contracts eligible to be entered into CPARS during the fiscal year, and a target of not less than 90% of past performance evaluations to be accomplished in CPARS within timeframes required in the Federal Acquisition Regulation (FAR). Accordingly, new and/or ongoing contracts will receive priority for completing past performance reporting over expired contracts, although contractor performance evaluations will be brought up to date as applicable. Recommendation 6: Determine whether domestic contract options are available for crucial repair parts that are identified as only being available from a foreign subcontractor. 12-P-0417 36 ------- Response: The EPA is conducting additional market research in accordance with FAR Part 10 to identify potential domestic sources prior to the re-compete of the spare parts contract. This contract is anticipated to be awarded prior to the end of FY 2012. Recommendation 7: Review the information in MATS for OIG Report No. 09-P-0087 and ensure it is accurate and current. Response: OIG Report No. 09-P-0087 contained findings and recommendations on several OAM contracts/orders. Below is the status on those past performance reporting requirements identified in MATS, as well as the RadNet delivery schedule. • Past performance evaluations for EP-W-05-012 were finalized in the system on 1/30/2009. • Past performance evaluations for 2008 and 2009 for Delivery Order 2 under EP-W- 07-076 were finalized in the system on 1/30/2009. • The 2010 past performance evaluation for Delivery Order 2 under EP-W-07-076 was finalized on 2/3/2012. • The past performance evaluation for Delivery Order 3 under EP-W-07-076 will be entered in April 2012 when the order has ended. • The expected completion date of installation or receipt of monitors is June 2012. Recommendation 8: Track the installation of the RadNet monitors against the revised schedule and use the contract requirements in recommendations 4 and 5 to hold the contractor accountable. Response: EPA concurs and the COR is working with the CO to enforce the terms and conditions of the contract including receipt of consideration for late deliveries. 12-P-0417 37 ------- Comments on T ext of Draft Report other than the Recommendations Pg. 3, Par. 2 and Table 1 The sentence just prior to Table 1 states that the contract for spare parts was awarded to the same contractor as for purchasing fixed monitors. While this is accurate, the EPA believes it is important to note that this was done for proprietary spare parts only, since there is no other known option. Currently, the proprietary parts are required for proper operation of the monitor. The EPA suggests the sentence read, "The contracts for fixed air monitoring stations and spare parts were awarded to the same contractor due to the proprietary nature of those spare parts." In Table 1, the total contract obligations under RadNet contract EP-D-10-085 for spare parts should be $1,405,913.29, in lieu of $8,517,587.00 as follows: Contract Delivery Order Date Amount EP-D-10-085 0001 September 21, 2010 $279,415.29 EP-D-10-085 0002 March 9, 2011 $127,495.00 EP-D-10-085 0003 March 19, 2011 $503,619.00 EP-D-10-085 0004 June 3, 2011 $495,384.00 $1,405,913.29 Pg. 3, Last Par. The second sentence says that the EPA increased sampling frequency following the Japanese nuclear incidents. For clarification, the EPA suggests that the sentence be revised to read, "In response to the Japan nuclear incident, the EPA increased sampling frequency for the milk and drinking water networks and increased analysis frequency for all networks to detect and measure radiation levels, and inform the public of any changes in those levels." Pg. 7, Par. 1 Since most of the late filter changes cited in this paragraph occurred while monitors were inoperable, EPA suggests that this fact be noted in the paragraph. Moreover, for an operating RadNet real-time monitor, reduced frequency of filter changes does not adversely affect either the hourly gamma data or their availability- nor does it adversely affect the gamma data obtained from laboratory analysis. One of the documents cited as containing filter change frequency is the 2008 RadNet Quality Assurance Manual (QAM). To clarify, this QAM does not describe operations or requirements of the RadNet real-time monitors that are the subject of this report; Section 5.3.4.1 of the QAM notes that the QAPP (for RadNet real-time monitors) is the applicable quality document for routine operations, not the QAM: "5.3.4.1 The RadNet air particulate monitoring program is 12-P-0417 38 ------- currently undergoing an expansion and upgrade to include near real-time gamma monitors. This long-term project of installing additional monitors began in 2006 and will not near completion until 2009. Please refer to the NAREL Quality Assurance Project Plan for Expansion of the RadNet Fixed Station Air Monitoring System to Include Near Real-Time Gamma Monitoring (RadNet/QAPP-1) for information. This manual [QAM] addresses all other aspects of the RadNet air monitoring program." EPA suggests that this QAM reference be deleted as a document associated with these monitors. Finally, for clarification, the twice weekly filter change references in the QAPP are intended to provide consistency in throughput at NAREL's analytical laboratory, not as operational requirements. As noted in the QAPP in sections 2.0 (page 14) and 7.0 (page 25), there is a distinct difference between the RadNet real-time and RadNet legacy (described in the QAM) monitors and the use of their filters. The filters from the RadNet real-time monitors actually measure radiation emitted from the filters themselves, and the monitors report the results continuously. In contrast, the filters from RadNet legacy monitors are the only way in which sampling/data can be collected from them. While filters from both types of monitors are sent to the analytical laboratory for analysis, the frequency of filter change for the RadNet real-time monitor does not affect the quality of the results from either the real-time data (from the filters) or the laboratory analysis of the filters. Suggested Revision: Pg. 7, Par. 1 Broken RadNet monitors and resulting late filter changes may impair this critical infrastructure asset. On March 11, 2011, at the time of the Japan nuclear incident emergency, 25 of the 124 installed RadNet monitors, or 20 percent, were out of service for an average of 130 days. In addition, 6 of the 12 RadNet monitors we sampled (50 percent) had gone over 8 weeks without a filter change, and 2 had gone unchanged for over 300 days because these monitors were broken. Unless EPA grants an extension, the repair services contract requires a service contractor to fix broken monitors within 14 days of EPA's notification that a monitor is out of service. The EPA QAPP refers to operators changing filters on fixed RadNet real-time monitors twice a week. Because EPA did not manage RadNet as a high-priority program this resulted in parts shortages, insufficient contract oversight, and contributed to the extensive delay in fixing broken monitors. In addition, broken RadNet real-time monitors and relaxed quality controls contributed to the filters not being changed timely. Out-of-service monitors and unchanged filters may reduce the availability and quality of critical data. As a result, EPA may not have sufficient data to determine levels of airborne radioactivity and the associated threat to public health and the environment EPA recognizes the expressed concern about RadNet station operability, and we have taken steps to address the issue more completely; however, the RadNet system was able to provide sufficient data to determine levels of airborne radioactivity during the weeks after the Fukushima nuclear power plant incident. EPA worked very closely with the interagency scientific and public health communities during the Fukushima response to properly characterize our findings and in the development of our public messages. Pg. 7. Par. 2 and Par. 3 As another point of clarification, filter changes for RadNet monitors are requested at a rate of twice per week for consistency with the legacy air monitoring program and for NAREL's 12-P-0417 39 ------- analytical laboratory work load planning purposes. As previously stated, the referenced QAM is not applicable to the RadNet real-time monitoring network. Twice weekly filter changes are not required for RadNet real-time monitors because changing filters less than twice weekly does not adversely affect data quality or availability. EPA also notes again that the spare parts contract is with the same contractor which makes the monitor. This is solely for procuring proprietary parts that are not available through other vendors. EPA suggests minor wording changes to clarify these points in the following paragraphs in redline-strikeout format. Suggested Revision: Pg. 7. Par. 2 EPA included terms and conditions in the RadNet repair and maintenance services contract to define the period of time for repair., and EPA established quality control standards for frequency of filter changes. In May 2008, EPA awarded a RadNet repair and maintenance service contract that requires the contractor to fix broken monitors within 14 days of being notified by the COR. EPA may permit an extension of this 14- day period for a specific repair for reasons including unavailability of government-furnished equipment, operator unavailability, and physical disruption of the site. EPA acquires the GFP from the contractor that was awarded through the proprietary spare parts contract. Suggested Revision: Pg. 7. Par. 3 For laboratory workload planning, EPA's QAM and QAPP estimates that, under routine circumstances, RadNet air station operators collect air particulate filters twice a week and mail the filters via the U.S. Postal Service to NAREL for analysis. When elevated levels of radioactivity are anticipated or known to exist, EPA may request RadNet station operators to increase the sampling frequency and use priority shipping. Pg. 9, Par. 1 The first paragraph on page 9 contains the sentences "EPA did not have the required parts to provide to the repair contractor, and would not for an extended period of time. Therefore, EPA did not notify the contractor that these monitors were in need of repair." EPA agrees with these sentences, and suggests that similar sentences be included in the last paragraph on page 9 for completeness to indicate why the COR took 120 to 421 days to notify the repair contractor. Pg. 9, Par. 2 In the second paragraph on page 9, the phrase "had not assigned operators for two monitors" is used. Since EPA cannot assign operators, we suggest that the phrase "had not been able to recruit replacement volunteer operators" be used instead. Pg. 10, Par. 1 EPA requests the first sentence on page 10 be revised to indicate that some repair requests had been made to the service contractor prior to the earthquake, and that the priority repairs were 12-P-0417 40 ------- made possible only by "cannibalizing" monitors in the process of being constructed. EPA suggests the first paragraph on page 10 be changed to read (changes highlighted in red): Suggested Revision: Pg. 10. Par. 1 "Eighteen of the 25 out-of-service monitors received EPA's priority attention only after the Japan nuclear incident. Although 7 out-of-service monitors had been scheduled for repair prior to the Japan incident, the remaining 18 out-of-service monitors received EPA's priority attention only after the incident The priority attention consisted of cannibalizing monitors under construction for their parts which were used to repair broken monitors. By April 8, 2011, the service contractor had completed repairs on all monitors (figure 4)." Pg. 11. Par. 2 Since the QAM is not an applicable reference for the RadNet real time monitors, EPA suggests removing references to it in the first sentence of this paragraph and the preceding bullets. Also, the context of the QAPP section referenced (laboratory analytical scheduling) should be noted in this sentence. EPA suggests the sentence read "The RadNet QAPP estimates twice-a-week filter changes for laboratory analytical scheduling purposes." Pg. 11. Last Par In the last paragraph on page 11, EPA recommends removing the statement that there is a requirement to change filters twice per week. Since changing filters less frequently than twice per week does not affect hourly gamma data quality or availability or laboratory analysis for gamma radiation, this does not adversely affect data completeness and does not potentially impair RadNet's ability to protect public health. EPA suggests the last paragraph on page 11 read as follows (changes highlighted in red). Suggested Revision: Pg. 11, Par. 2 Further, NAREL gave operators permission to wait up to 8 days between filter changes, despite the requirement to change the filters initial provisional request of twice a week in the QAM and QAPP. Some operators requested and NAREL gave permission to change filters once a week. Failure to follow RadNet quality assurance requirements may adversely affect data completeness and potentially impairs RadNet's ability to protect human health Pg. 12. Par. 2 The second paragraph of page 12 is correct. For clarification however, EPA requests that language be added to clarify that, even though there was not a spare parts contract in place, EPA was purchasing proprietary spare parts from the vendor using individual purchase orders. EPA suggests the following language be added to the end of this paragraph to reflect this: "Prior to award of the Spare Parts Contract, multiple stand-alone purchase orders totaling over $200,000 were placed with the manufacturer for proprietary parts not available from other vendors. These various orders included computers (LPU's), detector assemblies and other components subsequently included in the Spare Parts contract. The parts identified during these purchases provided input for development of the subsequent Spare Parts Contract." 12-P-0417 41 ------- Pg. 16, Par. 1 and Table 5 EPA suggests the following sentence be added as the second sentence to the first paragraph in this section on page 16: "Just prior to release of our draft report to OAR and OAM, EP-D-12- 003 was awarded on December 12, 2011 for RadNet Air Monitor Maintenance to Environmental Dimensions, Inc. This contract replaced EP-D-08-068 and does include disincentives for subpar performance as follows: A five (5) percent reduction on the invoiced labor amount for every one (1) percent slippage from the ninety-five (95) percent performance level." EPA also requests that Table 5 be updated to show this disincentive as a Yes. Pg. 17 (Planned Disincentives Not Included in Spare Parts Contract) EPA offers the following text to provide additional information for this section: The procuring CO, Rodney-Daryl Jones was interviewed by Marcia Hirt-Reigeluth, OIG/USEPA on May 23, 2011 and the only discounts mentioned were those dealing with volume discounts included on the parts schedule. The Procurement Initiation Notice (PIN) - Spare Parts Acquisition for RadNet Fixed Monitors contained a parts schedule which identified the parts and estimated quantities. A SOW was not provided with the PIN, this being a commodity acquisition. The program office did not provide a SOW or any document containing disincentives. A new contract will be established prior to the end of FY 2012 for RadNet Fixed Monitor Spare Parts addressing incentives/disincentives, monthly progress reports and current parts forecast incorporating historical monitor failure data, which was previously not available. The current contract EP-D-10-085 was established with a five year estimate of $1,369,482.00. Pg. 20, Par. 1 In the first paragraph on page 20, EPA requests that the second sentence be changed to include that the contract is now awarded rather than being a future contract. EPA suggests the sentence read: "The September 2011 version of the SOW for the RadNet service contract awarded in December 2011 contains language to address the proprietary issue." Pg. 20, Next to Last Par. EPA notes that the new spare parts contract will contain requirements for MPRs. Pg. 20, Last Par. EPA notes that the RadNet air monitor maintenance contract, EP-D-08-068 provided more than adequate reporting through the requirements contained in the SOW and detailed invoicing requirements. A monthly report of the repairs would only be a compilation of the previously submitted individual reports and would have added no additional value. EP-D-08-068 SOW Task 1.6 12-P-0417 42 ------- Within 7 days of completing on-site maintenance and repair of a monitor, the Contractor shall provide a written report to the COR describing; date of on-site service request; date of on-site service performance; name of service technician performing on-site maintenance and repair; specific monitor deficiencies reported to the Contractor; specific monitor deficiencies encountered by the Contractor; steps and procedures performed to return the monitor to fully functional and calibrated condition; itemization and cost of parts required to return the monitor to a fully functional and calibrated condition; itemization of billable hours related to the on-site maintenance and repair. Pg. 28. Table The table entry for compensation received for Modification 4 of delivery order 1 is incorrect. Two MABs were received, not LPUs. 12-P-0417 43 ------- Appendix E Distribution Office of the Administrator Assistant Administrator for Air and Radiation Assistant Administrator for Administration and Resources Management Acting Director, National Air and Radiation Environmental Laboratory Director, Office of Acquisition Management, Office of Administration and Resources Management Agency Follow-Up Official (the CFO) Agency Follow-Up Coordinator General Counsel Associate Administrator for Congressional and Intergovernmental Relations Associate Administrator for External Affairs and Environmental Education Audit Follow-Up Coordinator, Office of Air and Radiation Audit Follow-Up Coordinator, Office of Administration and Resources Management 12-P-0417 44 ------- |