Office of Inspector General Audit Report SUPERFUND Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls Report No. E1SKG7-08-5001-8100082 March 25,1998 ------- Inspector General Division Central Audit Division Conducting the Audit: Denver, Colorado Region Covered: Region 8 Program Office Involved: Office of Ecosystems Protection And Remediation Office of Technical and Management Services ------- UNITED STATES ENVIRONMENTAL PROTECTION AGENCY OFFICE OF INSPECTOR GENERAL _ : CENTRAL AUDIT DIVISION V^V J&WgfiXX* B-CH0mCES *i PBO^ 1445 ROSS AXENJS. SUITE 1:0; 913-551-7878 DALLAS TEXAS 75202-:-;:- FAX: 913-551-7837 2i--e-:s-eei' FAX 21--665-6589 999 18Tn STPEE" SU TE 50C DENVER C 0,0= ADC 80202-2-:5 March 25, 1998 FAX go^lsS MEMORANDUM SUBJECT: Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls Audit Report E1SKG7-08-5001-8100082 FROM: Bennie S. Salem &^~it^ v^J-£^~—. Divisional Inspector General TO: William P. Yellowtail Regional Administrator Region 8 We have completed our followup audit of the U. S. Environmental Protection Agency (EPA) Region 8's Superfund field sampling quality assurance activities. Attached is our report entitled Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality- Assurance Controls. We found that the Region needed to better define and document its data quality objectives, further strengthen controls over field sampling activities, improve its quality assurance training to its remedial project managers, and ensure increased oversight and improve documentation of oversight at Superfund sites. We met with the Region 8 Superfund Remedial Response program manager and supervisor and the Regional Quality Assurance program manager who generally concurred with our recommendations and agreed to begin corrective actions. We incorporated their comments as appropriate. Action Required In accordance with EPA Order 2750, you, as the action official, are required to provide us a written response to the audit report within 90 days of the final audit report date. For corrective actions planned but not completed by the response date, reference to specific milestone dates will assist us in deciding whether to close this report. This audit report contains findings that describe issues the Office of Inspector General (OIG) has identified and corrective actions OIG recommends. This report represents the opinion of OIG, and the findings contained in this report do not necessarily represent the final EPA position. Final determinations on matters in this report will be made by EPA managers in accordance with established EPA audit resolution procedures. In this audit, OIG did not measure the reviewed offices against all standards established by the National Contingency Plan. The findings contained in the report are not binding in any enforcement proceeding brought by EPA or the Department of Justice under Section RECYCLED ------- 107 of the Comprehensive Environment Response, Compensation, and Liability Act to recover costs incurred not inconsistent with the National Contingency Plan. We have no objections to the release of this report to any member of the public. This report contains no confidential business or proprietary information that cannot be released to the public. Should you or your staff have any questions regarding this report, please contact me at (913) 551-7831 or Jeff Hart, Audit Manager in our Denver office, at 312-6169. Please refer to the report number on all related correspondence. Attachment ------- Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls EXECUTIVE SUMMARY INTRODUCTION The Office of Inspector General conducted a followup audit of Region 8's Superfund field sampling quality assurance activities. In our January 1995 report, Implementing Controls Would Improve Region 8's Superfund Field Sampling Activities (E1SKG4-08-0045-5400034), we found that Region 8 needed to strengthen pre-field sampling controls, better train its remedial project managers (project managers), and ensure an appropriate level of oversight at Superfund sites. Our followup audit disclosed that similar problems continued to exist. The U. S. Environmental Protection Agency (EPA) requires that each region or major organization have a quality assurance program to ensure that activities generate data of known and adequate quality. EPA stated in its 1997 strategic plan that it "will continue to implement its mandatory Quality Assurance Program" to ensure high quality environmental data. OBJECTIVES Our general objective was to review Region 8's improved procedures and processes for review, approval, and implementation of field sampling quality assurance activities in the Superfund Remedial Program. Specifically, our objectives were to answer the following questions: * Is the Region properly using data quality objectives to support decisionmaking? * Are adequate controls in place to ensure that an approved sampling and analysis plan is in place prior to commencing field sampling activities? E1SKG7-08-5001-8100082 ------- Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls Do project; managers possess the skills necessary, through training, to approve sampling and analysis plans? Is Regional oversight of field sampling activities adequate to ensure compliance with sampling and analysis plans and reliable and defensible data? RESULTS IN BRIEF A Consistently Applied Data Quality Objectives Process Would Provide For Better Decisions Region 8 improved some of its field sampling quality assurance activities since 1995. However, the Region could more effectively support EPA's goal for high quality environmental data by better implementing its quality assurance program. Project managers could better define, document, and update data quality objectives to support environmental cleanup decisions. Also, Region 8's strengthened controls did not fully achieve their intended purposes to prevent untimely project plan submissions and undocumented plan approvals. Supervisors had not ensured that all parties responsible for field sampling activities attended quality assurance training. Project managers continued to perform limited oversight over contractors and did not routinely look to the Region 8 quality assurance group for assistance in reviewing project plans. Project managers could have more consistently applied the data quality objectives process as specified by EPA policy. Specifically, they should have better defined, documented, and updated data quality objectives to support their environmental cleanup decisions. Data quality objectives in EPA project plans were too broadly defined to ensure sufficient and appropriate data was gathered, and project managers accepted deficient contractor-prepared plans. Without evidence of decisions based on sound data, EPA is vulnerable to criticism and risks inadequate, inappropriate, or untimely sampling and cleanup. E1SKG7-08-5001-8100082 ------- Region 8 Needs to Further Strengthen Its Supeifund Field Sampling Quality Assurance Controls Accountability For Strengthened Controls Would Help Eliminate Planning Problems Training Would Strengthen Quality Assurance Skills Further Actions Could Ensure Appropriate And Well-Documented Oversight Region 8's strengthened controls did not fully achieve their intended purposes to prevent untimely project plan submissions and undocumented plan approvals. Some project managers continued to allow contractors and potentially responsible parties to submit late plans and neglected to ensure their plans included signed approval pages. Without clearly identified and approved plans, the Region could not ensure it (1) collected the necessary data to accurately identify pollution sources and assess risks, or (2) made the most efficient, effective, or scientifically defensible cleanup decisions. Region 8 agreed that project managers needed to attend mandatory quality assurance training to ensure they had the necessary skills to properly approve project plans. However, 13 (or 38 percent) of the project managers and other staff needing the training had not completed the two mandatory Regional courses. Management should have held project managers accountable for attending training, and the Regional quality assurance group needed to improve training effectiveness. Without well-trained project managers, Region 8 could not ensure project managers had the necessary skills to properly review and approve project plans. Region 8 had inadequate assurance that data collected during the remedial cleanup phase was reliable and defensible because project managers had not significantly changed their oversight practices to ensure compliance with project plans. Some project managers continued to perform limited oversight, relied too heavily on oversight contractors, did not use the Region 8 quality assurance group's assistance, and had not appropriately documented their oversight activities. Project managers had no procedures specifically describing their oversight and site management responsibilities and authority. Without appropriate and documented oversight, Region 8 could not ensure data was reliable and defensible. E1SKG7-08-5001-8100082 in ------- Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls RECOMMENDATIONS We recommend that the Regional Administrator (RA) develop Regional controls to ensure that Regional staff appropriately develop data quality objectives^ document the process in the project plans, and hold Superfund supervisors accountable for ensuring that project managers have actually reviewed and properly approved project plans. Further, the RA should direct supervisors to identify staff who have not completed required training and hold staff accountable for attending. The RA should also direct supervisors to hold project managers accountable for following existing Region 8 policy and determine whether additional oversight direction is needed. AGENCY COMMENTS AND OIG EVALUATION Region 8 generally concurred with our recommendations and presented an action plan that outlined how the Region planned to address our recommendations and generally satisfied the intent of our recommendations. We agree that Region 8 had improved its field sampling quality assurance controls since our last report. We also agree that the Region can further improve its quality assurance efforts in its Superfund field sampling planning. We reviewed Region 8's specific comments to our draft report and made some minor modifications to the final report as we determined necessary. E1SKG7-08-5001-8100082 IV ------- Region 8 Needs to Further Strengthen Its Super-fund Field Sampling Quality Assurance Controls TABLE OF CONTENTS Page EXECUTIVE SUMMARY i ABBREVIATIONS vii CHAPTERS 1 INTRODUCTION 1 Purpose 1 Background 1 ^ Scope And Methodology 4 Prior Audit Coverage 4 2 A CONSISTENTLY APPLIED DATA QUALITY OBJECTIVES PROCESS WOULD PROVIDE FOR BETTER DECISIONS 6 EPA Specifies Using The Data Quality Objectives Process 6 Managers Needed to Demonstrate Use of The Data Quality Objectives Process 8 Project Managers Accepted Deficient Plans 11 Conclusion 11 Recommendations 12 3 ACCOUNTABILITY FOR STRENGTHENED CONTROLS WOULD HELP ELIMINATE PLANNING PROBLEMS 13 EPA Requires Timely Reviewed and Approved Plans 13 Project Managers Did Not Properly Apply Additional Controls 14 Accountability Had Not Been Enforced 18 Conclusion 19 Recommendations 19 4 TRAINING WOULD STRENGTHEN QUALITY ASSURANCE SKILLS 21 EPA and Region 8 Require Quality Assurance Training 21 Staff Had Not Completed Required Training 22 Management Did Not Regularly Emphasize Quality Assurance Training 23 Project Managers Did Not Value Quality Assurance Training 24 E1SKG7-08-5001-8100082 ------- Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls Conclusion 25 Recommendations 26 5 FURTHER ACTIONS COULD ENSURE APPROPRIATE AND WELL-DOCUMENTED OVERSIGHT 27 Project Managers' Responsibilities Include Oversight 27 Oversight Activities Had Not Significantly Improved 28 Oversight Responsibilities and Authority Were Unclear 30 Project Managers Did Not Value Quality Assurance Assistance 31 Conclusion 32 Recommendations 33 EXHIBITS 1 THE DATA QUALITY OBJECTIVES PROCESS COMPARED TO THE SCIENTIFIC METHOD 34 2 SCOPE AND METHODOLOGY 35 3 EPA COMMENTS AND OIG EVALUATION 37 APPENDICES I EPA RESPONSE 38 H DISTRIBUTION 43 E1SKG7-08-5001-8100082 vi ------- Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls ABBREVIATIONS EPA U. S. Environmental Protection Agency OIG Office of Inspector General Project Manager Remedial Project Manager E1SKG7-08-5001-8100082 vii ------- Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls CHAPTER 1 INTRODUCTION PURPOSE The Office of Inspector General (OIG) conducted a followup audit of Region 8's Superfund field sampling quality assurance activities. Our general objective was to review Region 8's improved procedures and processes for review, approval, and implementation of field sampling quality assurance activities in the Superfund Remedial Program. Specifically, our objectives were to answer the following questions: *• Is the Region properly using data quality objectives to support decisionmaking? *• Are adequate controls in place to ensure that an approved sampling and analysis plan is in place prior to commencing field sampling activities? *• Do project managers possess the skills necessary, through training, to approve sampling and analysis plans? * Is Regional oversight of field sampling activities adequate to ensure compliance with sampling and analysis plans and reliable, defensible data? BACKGROUND The U. S. Environmental Protection Agency (EPA) requires that each region or major organization have a quality assurance program to ensure that activities generate data of known and adequate quality. EPA stated in its 1997 strategic plan that it "will continue to implement its mandatory Quality Assurance Program" to ensure high E1SKG7-08-5001-8100082 ------- Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls quality environmental data. The quality assurance program goal is to measure and minimize sources of error and to monitor conditions of sampling, storage, and transport. Quality assurance and quality control operate hand-in-hand to help managers meet EPA's quality assurance goal. Quality assurance is a management function that should provide policy and an integrated system of management controls for planning, implementation, and review of environmental data collection activities. Quality control is a system designed to measure the attributes and performance of an activity against specific standards and verify that they meet the standards. Quality control should be implemented at the project level and include such scientific precautions as calibrations and duplications. EPA Order 5360.1, Policy and Program Requirements to Implement the Mandatory Quality Assurance Program, dated April 1984, applies to all EPA programs including Superfund. It provides for each EPA organization to implement the requirements of EPA's quality assurance program. EPA Order 5360.1 requires data be of known quality and states that data are of known quality when data "...are thoroughly documented, such documentation being verifiable and defensible." EPA developed a "data quality objectives" process as its systematic planning tool to help ensure it consistently collected the quality of data needed to support technical decisions. The data quality objectives process is based on the scientific method and thus contributes to EPA's goal of sound science. EPA requires project plans to include a data quality objectives section to document the project's defined goal and planning outputs. The National Contingency Plan regulates the Superfund Remedial Program through Part 300 of 40 Code of Federal Regulations. Sections 300.420 and 300.430 require sampling and analysis plans that provide a process for E1SKG7-08-5001-8100082 ------- Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls obtaining environmental data of sufficient quality and -quantity to support decisionmaking. EPA must review and approve sampling and analysis plans at most Superfund sites. Sampling and analysis plans consist of: (1) a field sampling plan, and (2) a quality assurance project plan. A field sampling plan describes the number, type, and location of samples and the type of analysis required. The quality assurance project plan describes policy, organization, and functional activities necessary to develop adequate data. The data is used for planning and documenting a removal action, site evaluation, and hazard ranking system activities. EPA staff often use the terms "sampling and analysis plan," "field sampling plan," and "quality assurance project plan" interchangeably. Throughout this report, we have used the term "project plan" to refer to all these types of plans except when we have described a specific example. EPA's quality assurance program requires each organization that collects environmental data to implement an approved quality management plan. Region 8's Quality Management Plan, dated September 1996, is Region 8's statement of and commitment to quality assurance activities. It describes Region 8 quality management policies, objectives, principles, responsibilities, and accountability. Region 8's Office of Ecosystems Protection and Remediation and Region 8's Montana Office are responsible for Superfund cleanup and oversight. On-scene coordinators, site assessment managers, and remedial project managers (project manager) are responsible for ensuring field sampling integrity at their sites and ultimately report to the Superfund program managers located in Denver and Montana. Region 8's Office of Technical and Management Services has responsibility for the quality assurance program. The quality assurance group oversees the quality of data- generating activities throughout the Region. E1SKG7-08-5001-8100082 ------- Region 8 Needs to Further Strengthen Its Super-fund Field Sampling Quality Assurance Controls SCOPE AND METHODOLOGY We performed a followup audit of EPA Region 8's Superfund field sampling quality assurance activities to determine whether Region 8 improved its controls over its field sampling activities as a result of our report, Implementing Controls Would Improve Region 8's Superfund Field Sampling Activities, E1SKG4-08-0045- 5400034, dated January 27, 1995. We performed our audit in accordance with Government Auditing Standards (1994 Revision) issued by the Comptroller General of the United States. Other than the issues discussed in this report, no other significant issues came to our attention that warranted expanding the scope of our review. The scope of our 1995 report and our followup audit work was limited to the Superfund Remedial Program. See Exhibit 2 for additional information about our scope and methodology. PRIOR AUDIT COVERAGE The only relevant prior audit coverage was the report cited above. We have summarized relevant portions of the report below. We found that project managers inconsistently approved project plans and did not denote final, approved plans. Project managers previously allowed contractors to submit project plans for review and approval without providing 30 days for EPA review and approval prior to a sampling event. At the time, Region 8 allowed verbal approvals. Consequently, project managers had not documented their review and approvals in project plans. Some project managers prepared approval letters, but did not attach the letters to the approved plan or identify the approved plan in the letter. Our 1995 report disclosed that project managers recognized they had not fully complied with mandatory training requirements within specified timeframes or understood quality assurance requirements even after having attended Regional quality assurance courses. Project managers E1SKG7-08-5001-8100082 ------- Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls identified weaknesses in training and requested better presentations to include identifying quality assurance experts for project managers to contact. None of the project managers had completed all three mandatory quality assurance courses required by the Region as of January 1994. Also, the Region needed to better document training attendance. We further found that project managers relied on contractors to conduct sampling with limited oversight, did not always adequately oversee activities to ensure proper contractor performance, and did not enhance their own experience by observing site activities. Project managers had not used field audits to help them oversee their sites. In response to our 1995 report, the Region instituted strengthened controls over its field sampling activities by distributing two internal policy memorandums. The first memorandum to all Region 8 office directors instructed them to use an existing Region 8 sampling brokering system for all analytical work. Project managers were to attach approved and signed project plans to their requests for analytical services. Requests were required 1-6 weeks before a sampling event. The Region agreed that project managers and contractors should have followed a 30-day approval rule and suggested that staff include the requirement in contractor project planning schedules. The second memorandum instructed project managers to approve project plans in writing prior to field sampling and include the approval pages within the document. The Region further directed its project managers to take the mandatory quality assurance training. The memorandum advised project managers of other related quality assurance and program training available to enhance their skills. The memorandum emphasized contractor oversight and provided suggestions of where to get additional technical expertise to review project plans, how to use field audits to fulfill oversight responsibilities, and how to document site decisions. E1SKG7-08-5001-8100082 ------- Region 8 Needs to Further Strengthen Its Super-fund Field Sampling Quality Assurance Controls CHAPTER 2 A CONSISTENTLY APPLIED DATA QUALITY OBJECTIVES PROCESS WOULD PROVIDE FOR BETTER DECISIONS Project managers could have more consistently applied the data quality objectives process. EPA policy and guidance specifies using the data quality objectives process to support site decisionmaking. Project managers should have better defined, documented, and updated data quality objectives to support their environmental cleanup decisions. Region 8's project plans did not provide evidence that Region 8 used the data quality objectives process or another systematic planning approach to support decisions. Project managers stated they routinely reviewed contractors' project plans, but our review indicated they routinely accepted deficient plans. Without evidence of decisions based on sound data, EPA is vulnerable to criticism by Congress, states, potentially responsible parties, and the public for site mismanagement and unnecessary expenditures. Additionally, it risks inadequate, inappropriate, or untimely sampling and cleanup. EPA SPECIFIES USING THE DATA QUALITY OBJECTIVES PROCESS The Deputy Administrator directed in a May 1984 memorandum, Data Quality Objectives, that "Beginning in September 1984, data quality objectives should be an integral part of each significant new data collection activity." The Deputy Administrator further emphasized that "each National Program Manager is responsible for establishing data quality acceptance criteria (i.e., data quality objectives) for all of their projects...." A subsequent Deputy Administrator continued to focus on the importance of data quality objectives in a November 1986 memorandum, Agency Institutionalization of Data E1SKG7-08-5001-8100082 ------- Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls Quality Objectives. This Deputy Administrator directed that EPA implement the data quality objectives process. The November 1986 memorandum stated that the data quality objectives process was "sound management planning" and a key element of EPA's quality assurance program. The memorandum also stated that data quality objectives are explicit statements of the quality of data needed to support a decision. The Deputy Administrator stated that institutionalization of this process would save money and improve data quality. General EPA Quality Assurance Division guidance and specific Superfund program guidance provide detailed instructions for project managers to implement the data quality objectives process to support decisionmaking. The Quality Assurance Division's September 1994 Guidance for the Data Quality Objectives Process (QA/G-4) details a seven-step process to help project managers collect appropriate data. The data quality objectives process is a systematic planning tool based on the scientific method for determining the type, quantity, and quality of data that will be sufficient and appropriate for the data's intended use. Although project managers are not required to document each of the seven steps in project plans, the plans must adequately explain the results of the process. The Office of Emergency and Remedial Response also specifies using the process for all Superfund cleanups in its September 1993 Data Quality Objectives Process for Superfund (EPA 540- R-93-071). Region 8's Quality Management Plan requires project managers to develop and adequately document data quality objectives in their project plans that are appropriate for the site circumstances. w Because the data quality objectives process is based on the scientific method, it "...improves the legal defensibility of site decisions by providing a complete record of the decision process and criteria for arriving at conclusions." The scientific method provides principles and processes necessary for scientific investigation and a means for E1SKG7-08-5001-8100082 ------- Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls independent observers to reach the same conclusion. See Exhibit 1 for a more detailed description of how the data quality objectives process builds upon the scientific method. Data quality objectives determine quality control requirements for data collection, sampling, and analysis. Data quality objectives provide the purpose and focus as well as the qualitative and quantitative criteria that: (1) define the most appropriate type of data to collect, and (2) specify the limits on decisionmaking errors to identify acceptable risk. The seven steps in the data quality objectives process are shown in Exhibit 1. In accordance with the September 1994 data quality objectives guidance, data quality objectives must be continually assessed and revised as site circumstances change. Also, the Quality Assurance Division's August 1994 EPA Requirements for Quality Assurance Project Plans for Environmental Data Operations (QA/R-5) requires project managers to review project plans, at least annually, for long duration projects or multi-year projects. For example, Superfund sites sometimes take 10 years or more to clean up. During such a lengthy cleanup process, much data is collected, many decisions are made, and public interests may change. In addition, new technology or new information on health risks may become available. Finally, site conditions may change as a result of unanticipated outcomes of other site decisions. MANAGERS NEEDED TO DEMONSTRATE USE OF THE DATA QUALITY OBJECTIVES PROCESS Project managers did not use or adequately document the use of the data quality objectives process in making field sampling decisions. The data quality objectives for specific projects did not adequately support decisionmaking because project managers had not adequately: (1) defined data quality objectives, (2) appropriately documented data E1SKG7-08-5001-8100082 ------- Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls quality objective results in their project plans, and (3) updated project plans as site circumstances changed. Although project managers stated they applied the data quality objectives process, the objectives in their project plans were too broadly defined to ensure sufficient and appropriate data was gathered. Also, prior to 1992, the quality assurance manager approved project plans but had not identified deficient data quality objectives or ensured the data quality objectives were well documented. We examined 36 project plans and determined that the data quality objectives in 8 plans (22 percent) were adequate, while 28 (78 percent) were deficient. The eight adequate plans were all prepared prior to the issuance of the 1994 data quality objectives guidance. These eight plans appeared to follow the EPA data quality objectives process and appropriately documented the specific project needs by explaining the results of the planning process. The 28 deficient project plans included data quality objectives that were broadly defined and were only task or project objectives which did not directly contribute towards specific site decisionmaking. While the majority of the 28 deficient plans were prepared prior to issuance of the 1994 guidance, 8 of the 28 were prepared after guidance issuance. As an example of the deficiencies, we noted the data quality objectives for the California Gulch Superfund site plan clearly stated a need to conduct a sampling study. However, the objectives did not identify the point at which enough information would be collected and a decision could be made. Appropriate data quality objectives for this project should include statements of the problem (i.e., the project/task objectives); specific inputs needed; a decision statement (i.e., a point at which remedial action should be taken); risks associated with the decision, if the critical sample results were too close to the action level; and an optimal design. E1SKG7-08-5001-8100082 ------- Region 8 Needs to Further Strengthen Its Super-fund Field Sampling Quality Assurance Controls Region 8 quality assurance staff told us that project plans they reviewed over the last 2 years generally either did not specify or incorrectly specified limits on acceptable decision errors. According to the Region 8 quality assurance manager and the staff statistician, project managers generally had not adequately identified limits on decision errors (data quality objective step six). Project managers had not accurately calculated the total error and could not have known the quality of the collected data. While some plans identified an acceptable laboratory error rate, they did not generally identify an acceptable field error rate. The plans generally implied that the laboratory error rate was equal to the total error rate. Therefore, the plans did not demonstrate that implementation would result in data of known quality. Project managers did not update project plans to account for changing site circumstances. All of the plans should have contained a discussion of how the data quality objectives evolved. However, only 1 of the 36 plans was updated with this information. Plans for sites that consisted of multiple operable units or phases referenced a generic quality assurance project plan that was developed many years before the most current sampling event. For example, for the Silver Bow Creek/Butte, Montana site, a 1996 remedial design/remedial action sampling event referred to the overall Site Remedial Investigation/ Feasibility Study Quality Assurance Project Plan dated February 1992. This practice may have been considered efficient for multiple remedial investigation/feasibility study sample events. Using the 1992 project plan may also have saved time and resources to prepare a new quality assurance project plan for each event. However, multi- year plans should be evaluated annually because the old data quality objectives may no longer be relevant. In this case, the 1996 sampling event was related to remedial design/remedial action activities that are generally significantly different from remedial investigation/feasibility study objectives. E1SKG7-08-5001-8100082 10 ------- Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls PROJECT MANAGERS ACCEPTED DEFICIENT PLANS Although five of seven project managers stated that they understood and applied the data quality objectives process, the project plans we reviewed showed that project managers did not understand what they were reviewing or chose to accept plans with deficient data quality objectives sections. Generally, project managers used contractors to prepare the project plans we reviewed, including the data quality objectives section. Project managers stated that they provided the contractors with EPA guidance and requirements documents to use when the contractor prepared the project plans. Six of seven project managers explained that they looked for data quality objectives sections in their project plans during their reviews of the plans. However, although contractors included the required data quality objectives section in project plans, our review of project plans showed that project managers had not identified basic deficiencies in the data quality objectives sections of the project plans or ensured that any data quality objectives steps taken were adequately documented in the project plans. Some project managers did not understand that the data quality objectives process could be modified to accommodate site-specific needs. Some project managers stated they believed the seven-step data quality objectives process was unnecessary and inflexible. They explained that not all seven data quality objective steps needed to be completed for every data collection activity but did not understand that not completing all steps was an acceptable practice, if reasons for omitting steps were documented. Instead, they viewed the process as inflexible and not tailored to site-specific needs and phases of cleanup. CONCLUSION Region 8 unnecessarily risked inadequate or inappropriate sampling and cleanup because it did not use or adequately document application of the data quality objectives process. Without adequate planning, EPA may waste resources E1SKG7-08-5001-8100082 11 ------- Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls developing sampling plans that are never used, collecting unnecessary samples, delaying cleanup, or cleaning up sites beyond what is cost effective. While we believe project managers should complete all seven steps, the process is flexible and should be tailored to the significance of the sampling or other site decision to be made. The Region 8 Quality Management Plan recognizes the process does not have to be complicated and laborious. Without proper documentation to support project managers' decisions, EPA is vulnerable to criticism by Congress and other outside parties. EPA must be able to clearly demonstrate that it has developed data of known quality, properly assessed risks, and that its decisions are based on the best scientific information available. To accomplish this, project managers must better define and document data quality objectives to adequately support their decisions. RECOMMENDATIONS We recommend the Regional Administrator: 2-1. Develop Regional controls to implement an effective data quality objectives process. Controls should be designed to ensure that Regional staff appropriately develop data quality objectives and adequately document the process in the project plans. 2-2. Direct project managers to review data quality objectives periodically throughout the cleanup process and replace or modify project plans to account for changing site circumstances. 2-3. Direct project managers to reject contractors' project plans that do not include an acceptable data quality objectives section. 2-4. Include in the controls for Recommendation 2-1, steps to ensure that contractor-prepared project plans include adequate data quality objectives. E1SKG7-08-5001-8100082 12 ------- Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls CHAPTER 3 ACCOUNTABILITY FOR STRENGTHENED CONTROLS WOULD HELP ELIMINATE PLANNING PROBLEMS Region 8's strengthened controls did not fully achieve their intended purposes to prevent untimely project plan submissions and undocumented plan approvals. EPA regulations and current Regional policy require timely and approved plans. However, some project managers continued to allow contractors and potentially responsible parties to submit plans late without consequence, and some project managers neglected to ensure their plans included signed approval pages. Senior managers allowed staff to disregard improved controls developed in response to the 1995 audit without being held accountable for results. Therefore, Region 8 could not be certain that project managers had adequate time to review plans or that proper sampling was done in accordance with approved plans. Without clearly identified and approved plans, the Region could not ensure it collected the necessary data to accurately identify pollution sources and assess risks. Also, the Region could not ensure it made the most efficient, effective, or scientifically defensible cleanup decisions. EPA REQUIRES TIMELY REVIEWED AND APPROVED PLANS EPA regulations and guidance and Region 8 policy require approved project plans and records that are defensible and verifiable. EPA Region 8 policy also specifies plans be timely. The Superfund National Contingency Plan regulations and Region 8 policy require that EPA review and approve all project plans. EPA Order 5360.1 requires written records that are defensible and verifiable. The Order holds program managers responsible and requires that information withstand reasonable challenges related to accuracy or truthfulness and results that can be substantiated. EPA Requirements for Quality Assurance E1SKG7-08-5001-8100082 13 ------- Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls Project Plans for Environmental Data Operations (QA/R- 5) states that plans must be approved prior to initiating site work. Regional policy memorandums specified that project managers should ensure they receive project plans in advance of sampling activities to allow adequate time for thorough reviews. The policy memorandums instituted 30 days as a standard. The Region provides for accountability for fulfilling quality assurance requirements through performance agreements. Superfund remedial performance agreements for staff and supervisors require adherence to quality assurance policies, procedures, guidelines, and regulations. Specifically, project managers must comply "...with data quality assurance policies, procedures, and guidelines to assure that data collected are of the quality and type needed to support decisions." Supervisors' job elements require them to establish, implement, and maintain a quality assurance program and ensure "...compliance with all quality assurance related Agency policies, procedures, and regulations." PROJECT MANAGERS DID NOT PROPERLY APPLY ADDITIONAL CONTROLS Although Region 8 strengthened its controls in response to our 1995 report, the controls did not fully achieve their intended purposes to prevent untimely submissions and undocumented plan approvals. Although the Region implemented a modified sample brokering system to help eliminate untimely plan submissions, some project managers did not use the control effectively. Also, project managers did not always follow Regional policy instituted in 1995 requiring approval signature pages in their plans. Some project managers neglected to ensure their plans included signed approval pages. Six of seven project managers did not use the Region 8 quality assurance group's modified sample brokering system effectively. The Region implemented a modified sample E1SKG7-08-5001-8100082 14 ------- Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls brokering system in August 1995 that required the sample broker to ensure a project manager had an approved project plan attached to the project manager's request for analytical services. Although the sample brokering system's primary purpose was to identify and coordinate Region 8's internal analytical service requests, the system also provided a control to help eliminate untimely project plan submissions. According to the quality assurance staff, the premise was to have project managers better plan all phases of their sampling activities. By working backwards, project managers could determine when they needed their contractors, other federal agencies, or states to submit the project plans for EPA's review. Project managers knew that Regional policy required them to request analytical services approximately 1-6 weeks prior to sampling and that approved project plans had to be attached to their request for analytical services. With this knowledge, project managers could better project timelines. However, project managers circumvented this control by not appropriately using the sample brokering system either to obtain needed analytical services or provide notification of external analysis. The project managers allowed their contractors, other federal agencies, and states to request sampling analysis outside the Regional sample brokering system. Project managers should have provided the sample broker copies of approved project plans and notified the sample broker of internal and external laboratory analysis. Five of seven project managers explained that they use a tiered approach when managing their contractors and based their concerns on their level of trust with their contractors. If a project manager had used a contractor for several years and was comfortable with the contractor's expertise, the project manager would not require the contractor to comply with the 30-day approval rule. However, as emphasized in our 1995 report, project managers increased the risk of inappropriate sampling because contractors may realize that a final draft plan submitted just prior to a sampling event E1SKG7-08-5001-8100082 15 ------- Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls may not receive a thorough review due to the late submission. Four of seven project managers stated they did not unconditionally require their contractors and potentially responsible parties to submit plans 30 days in advance. Project managers said they were not concerned about having 30 days to review plans because they were comfortable with the contractors' and potentially responsible parties' levels of experience and expertise. They explained they held discussions throughout the project and believed all parties were in agreement and the project plans would contain the needed information. Also, project managers stated they were not concerned that site activities had significantly changed to warrant a 30-day review period. However, the Region wasted money and time on at least one inappropriate sampling plan that may have been prevented had the Region required the contractor to follow the 30-day requirement. The project manager canceled a sampling event and rejected a contractor-prepared project plan for the Summitville, Colorado Superfund site because the plan was inadequate. However, the project manager had not received the project plan until the day before the sampling event and did not review the plan until that time. Although the contractor never implemented the plan, the Region was still obligated to pay for it as a deliverable product. While the project manager prevented the contractor from implementing the inadequate sampling plan, Region 8 paid about $28,000 for this plan that it could not use. In addition, Regional staff prepared a "last-minute" plan for the sampling event. The $28,000 did not include Regional staff costs. Some project managers had not diligently implemented the Region's approval policy and continued to produce plans that did not demonstrate adequate review and did not have approval signature pages included within the plan. We E1SKG7-08-5001-8100082 16 ------- Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls reviewed 8 of the 36 project plans in our sample that were developed and implemented after our 1995 report. Although we reported the lack of approval signatures in 1995, we found that 5 of the 8 project plans (63 percent) still were not signed by the project managers and only 1 of the 8 project plans received a quality assurance review. Of the 5 unsigned plans, 4 (80 percent) did not include approval signature pages within the plan, while 1 (20 percent) had a signature page that was not completed. The Region researched its files and found 2 of the 4 missing approvals after we issued our official draft report. However, neither of the 2 approvals were included in the project plans as required by Regional policy. In addition, the Region approved 1 of the 4 project plans' missing approvals based on our draft report findings. Project managers had not ensured that all plans met the requirement for approval signature pages. Although only three of seven project managers stated that they had seen the Region's 1994 policy memorandum requiring signature pages, all seven explained that they knew of the approval requirement. However, we found that they did not consistently review their contractor-prepared project plans to ensure a signature page was included. Without signature pages, we could not easily confirm that plans had been reviewed or approved by the project manager. Also, as we reported in 1995, because there is often more than one version of a particular plan, project managers or others would find it difficult to determine which version was approved for contractor use for sampling without approval documentation having been included in the plan. E1SKG7-08-5001-8100082 17 ------- Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls ACCOUNTABILITY HAD Project managers disregarded supervisory suggestions to NOT BEEN ENFORCED incorporate the 30-day review period into project planning schedules and as contract deliverable language for new contractor work. Project managers continued to allow contractors and potentially responsible parties to submit untimely project plans to project managers for review, comment, and approval without consequence. Project managers did not include language in contracts specifying the 30-day advance plan submission. Although project managers had penalty provisions available in consent decrees and other contractual documents, some project managers had not used these authorities to ensure timely submissions. As a result, project managers did always not hold contractors or potentially responsible parties accountable. Four of seven project managers ignored the 30-day approval requirement because they did not believe the requirement was always necessary. Project managers explained that because they had seen previous iterations of the project plans, the final draft was usually not a surprise. They also stated that sometimes the Region requested a contractor to prepare a project plan in less than 30 days to meet urgent requirements at a site. They stated that neither project managers nor contractors should be held accountable for meeting the 30-day requirement in such cases. However, supervisors had not confirmed that project managers met project plan review and approval requirements. Although Superfund remedial office supervisors' and staffs performance agreements required quality assurance compliance, supervisors did not ensure staff complied with all Regional requirements and staff did not meet all quality assurance requirements. Remedial office supervisors stated they empowered their staff to perform their duties as "they see fit" and explained that project managers were professional staff tasked with many competing priorities. Supervisors had notified their staff of the Region's policy requiring documented approvals within project plans and E1SKG7-08-5001-8100082 18 ------- Region 8 Needs to Further Strengthen Its Super-fund Field Sampling Quality Assurance Controls said they believed staff were complying. One supervisor said that unless he heard of a major problem, he did not believe there was a problem. As a result, unless notified of problems, supervisors did not consciously evaluate project managers' approval actions to ensure they were complying with the quality assurance requirements. CONCLUSION Region 8's strengthened controls did not fully achieve their intended purposes to prevent untimely project plan submissions and undocumented plan approvals. The Region issued a revised policy but did not ensure that the policy was implemented. As a result, little actually changed since our 1995 review. Without the required approval signature page, the Region could not be certain that the correct plan version was used and that sampling was done in accordance with the most current, approved plan. As a result, the Region could not ensure it collected all the necessary data to accurately identify pollution sources and assess risks. Also, the Region could not ensure it made the most efficient, effective, or scientifically defensible sampling and cleanup decisions. The Region needs to ensure that project managers comply with the revised Regional policy for timely submissions and written approval. Top'management needs to hold staff accountable for properly approving plans. Although we agree that staff should be empowered to perform their duties without undue supervisor scrutiny, supervisors must ensure that staff meet EPA and Regional policy requirements. Top management also needs to hold contractors and potentially responsible parties accountable for submitting timely plans. RECOMMENDATIONS We recommend the Regional Administrator: 3-1. Hold Superfund supervisors accountable for ensuring that project managers timely and properly E1SKG7-08-5001-8100082 19 ------- Region 8 Needs to Further Strengthen Its Super-fund Field Sampling Quality Assurance Controls review and approve project plans prior to sampling. 3-2. Implement existing penalty policies regarding late deliverables when a pattern of delay becomes evident. 3-3. Hold Superfund supervisors accountable for accurately evaluating project managers' quality assurance compliance in their performance appraisals. E1SKG7-08-S001-8100082 20 ------- Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls CHAPTER 4 TRAINING WOULD STRENGTHEN QUALITY ASSURANCE SKILLS Region 8 agreed that project managers needed to attend mandatory quality assurance training to ensure they had the necessary skills to properly approve project plans. However, 13 (or 38 percent) of the project managers and other staff needing the training had not completed the two mandatory Regional quality assurance training courses. Regional policy prescribes two mandatory quality assurance courses for project managers to ensure they gather sound data. The Superfund program managers had not regularly emphasized the importance of quality assurance training or ensured project managers attended. Some project managers believed the training was ineffective and therefore a low priority. Management should have held project managers accountable for attending training, and the Regional quality assurance group needed to improve training effectiveness. Without well-trained project managers, Region 8 could not ensure project managers had the necessary skills to properly review and approve project plans. EPA AND REGION 8 REQUIRE QUALITY ASSURANCE TRAINING EPA and Region 8 policies require certain staff to take mandatory quality assurance training. EPA Order 5360.1 and Region 8's 1996 Quality Management Plan require all individuals (staff and their supervisors) directly involved in data collection activities to have a working understanding of quality assurance and quality control activities. Specifically, Region 8's Quality Management Plan requires training for all project managers with environmental data operation responsibilities including planning, design, collection, or assessment activities. The Quality Management Plan further suggests that project managers' immediate supervisors I attend the training. Region 8's training is provided by the quality assurance group. E1SKG7-08-5001-8100082 21 ------- Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls Regional policy prescribes two mandatory quality assurance courses for project managers to ensure they gather sound data. The mandatory courses include a 2-day, 12-hour EPA course, entitled Regional Quality Assurance Planning Requirements; and a 1-day, 6-hour EPA course, entitled Data Quality Objectives. In addition, individuals are required to take a refresher course once every 2 years. The Region 8 Quality Management Plan states that these training requirements were effective on January 19, 1996. Affected staff had until September 30, 1996, to complete the required quality assurance training or lose their quality assurance plan approval authority. Project managers must take mandatory training for three reasons: (1) to comply with EPA and Regional policy, (2) to maintain their project plan approval authority, and (3) to enhance their skills to serve as project managers. STAFF HAD NOT COMPLETED REQUIRED TRAINING Although Region 8 Superfund supervisors notified and encouraged staff to take mandatory quality assurance training, 13 Region 8 Superfund remedial staff had not completed the required training. Thirty-five remedial staff (nine in the Montana office) were subject to the mandatory quality assurance training including Superfund remedial program managers, supervisors, project managers, and Senior Environmental Employees. However, 13 of the 35 staff (38 percent) had not completed the mandatory quality assurance training including the 2 Superfund remedial program managers, 9 project managers, and 2 Senior Environmental Employees. Seven of the 13 (54 percent), including the 2 program managers, had not taken any quality assurance training. Although the Regional Quality Management Plan does not require that the two Superfund remedial program managers take quality assurance training, Region 8 agreed that "...some familiarity with QA [quality assurance] E1SKG7-08-5001-8100082 22 ------- Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls requirements is important for managers...." Having the two Superfund program managers attend quality assurance training would enhance their ability to oversee the Superfund program and send an important message to staff that quality assurance is essential and valuable in meeting program goals. Nine project managers who had not completed training by September 30, 1996, were out of compliance with Regional policy and should have had their project plan approval authority revoked after that time. Without proper training, these nine project managers could not ensure they possessed the skills necessary to appropriately approve project plans. However, at least two of the project managers continued to approve plans. Region 8 should have revoked the signature authority for the nine project managers until the project managers had completed training. At a minimum, supervisors should have monitored these project managers' activities. One supervisor was aware that one project manager had not completed training and assigned the "untrained" project manager to work with a project manager that had completed all training requirements. The "untrained" project manager completed training in February 1997 and had not approved project plans between September 30, 1996, and February 1997. However, the remaining project managers had not been individually monitored. Without proper training, project managers could not ensure they possessed the skills necessary to approve project plans and, therefore, had not fulfilled their responsibilities as project managers. MANAGEMENT DID NOT REGULARLY EMPHASIZE QUALITY ASSURANCE TRAINING According to quality assurance staff, Region 8's two Superfund program managers had not regularly emphasized the importance of quality assurance training to staff by' identifying those staff needing training and had not ensured all staff attended the training. To fulfill Regional quality assurance training requirements, the quality assurance E1SKG7-08-5001-8100082 23 ------- Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls program manager must first identify staff requiring training. In a December 9, 1996, memorandum to all Region 8 "Program Directors," the quality assurance program manager requested a list of staff requiring quality assurance training to be submitted by January 15, 1997. Although one of the two Superfund program managers had responded in a previous year to a similar request, neither responded to the December 1996 request that would have identified 1997 training needs. Quality assurance staff said their perception, based on the Superfund program managers' unresponsive actions, was that Superfund program managers did not consider quality assurance important. Quality assurance staff were concerned that project managers had reached the same conclusion. PROJECT MANAGERS DID NOT VALUE QUALITY ASSURANCE TRAINING Project managers wanted better quality assurance training, but did not value the current Regional training offered because they believed the training was ineffective and, therefore, a low priority. Most of the project managers we interviewed expressed dissatisfaction with the current training provided by the Region 8 quality assurance group. In general, training course evaluations suggested: (1) providing more coverage of where to get more information, (2) relating quality assurance to the program and not the program to quality assurance, (3) providing laboratory tours to enhance quality assurance understanding, and (4) discussing more "horror stories" to make a point of quality assurance importance. Some project managers said the quality assurance training was too general and needed to be more program specific. They explained that some examples used in training did not accurately depict the problems they often encountered at their sites because their sites were at different cleanup phases than the examples used in training. Also, project managers criticized that presentations were often read from E1SKG7-08-5001-8100082 24 ------- Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls overheads and not presented in a manner that would have enhanced an inexperienced project manager's knowledge. Some of the Regional quality assurance training, including the data quality objectives course, did not effectively convey the underlying quality assurance concepts that would help project managers better oversee their sites. Project managers stated that Region 8 quality assurance group trainers had not linked quality assurance to their program areas in a manner that enhanced their quality assurance understanding. The Regional quality assurance program manager acknowledged that the group was trying to improve its training presentations. He stated that he had met with staff to discuss alternative presentation approaches that would provide clearer and more relevant information to program staff. He had also met with other Regional program managers to determine specific staff needs and planned to meet with other managers to assess training needs. CONCLUSION Thirteen (38 percent) of Region 8's Superfund remedial program staff had not completed mandatory quality assurance training, 9 of which were project managers with the responsibility to review and approve project plans. Without well trained project managers, Region 8 could not ensure project managers had the necessary skills to properly review and approve project plans. We believe a direct correlation exists between senior management emphasizing the importance of quality assurance and project managers attending training. When the two Superfund program managers did not respond to the quality assurance training request, the message delivered to the quality assurance group and, possibly to project managers, was that quality assurance is unimportant. The Region needed to ensure attendance policies are adhered to by all affected staff and that training is effective. E1SKG7-08-5001-8100082 25 ------- Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls RECOMMENDATIONS We recommend the Regional Administrator: 4-1. Direct the Superfund remedial program managers to identify staff who have not completed required training and hold staff accountable for attending. 4-2. Involve program and project managers in the design and timing of quality assurance training. 4-3. Direct the quality assurance staff to routinely solicit customer comments from program staff and evaluate comments to improve services. 4-4. Direct Superfund program staff to promote quality assurance through initiatives such as inviting quality assurance staff to attend Superfund program staff meetings to explain the benefits of quality assurance to the Superfund program. 4-5. Obtain training for the quality assurance staff in presentation techniques to help quality assurance staff better articulate how quality assurance adds value. E1SKG7-08-5001-8100082 26 ------- Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls CHAPTER 5 FURTHER ACTIONS COULD ENSURE APPROPRIATE AND WELL-DOCUMENTED OVERSIGHT Region 8 had inadequate assurance that data collected during the remedial cleanup phase was reliable and defensible because project managers had not significantly changed their oversight practices to ensure compliance with project plans. Regional policy, project managers' position descriptions, and performance agreements outline oversight responsibilities and good management practices. However, some project managers continued to perform limited oversight, relied too heavily on oversight contractors, and did not use the Region 8 quality assurance group's assistance. Also, project managers had not appropriately documented their oversight activities. Project managers had no procedures specifically describing their oversight and site management responsibilities and authority. Project managers needed more specific direction and clearer authority. When project managers rely too heavily on contractors and do not observe contractor activities, they cannot ensure contractors perform as intended, are fully knowledgeable of site activities, or enhance their own experience. Without appropriate and documented oversight, Region 8 could not ensure data was reliable and defensible. PROJECT MANAGERS' RESPONSIBILITIES INCLUDE OVERSIGHT Regional policy outlines project manager oversight responsibilities and good management practices. As a result of our prior review, the Region issued a 1995 policy memorandum that provided some general suggestions to project managers on contractor oversight and identified other technical personnel available to assist them. The policy memorandum emphasized that oversight of contractors was one of project managers' most important responsibilities and that project managers should use available Regional expertise to review workplans. The E1SKG7-08-5001-8100082 27 ------- Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls memorandum also instructed project managers to evaluate contractor performance, such as through field audits. The policy memorandum also directed project managers to document activities in "records of communication" as a good management practice and to periodically observe field sampling activities. Project manager oversight responsibilities had not changed since our 1995 review and are addressed in generic position descriptions and performance agreements. Project manager position descriptions at General Schedule levels 9-13 still included several statements applicable to oversight activities, including: (1) ensuring technical soundness of remedial efforts accomplished by contractors, states, and potentially responsible parties; and (2) directing, overseeing, managing, and evaluating contractor actions. Generic performance agreements specifically required project managers to oversee compliance with orders and decrees and take appropriate action in cases of noncompliance; monitor contractor performance through invoices and supporting documentation; maintain appropriate records; communicate clearly, concisely, timely, and responsively with colleagues and managers; and prepare written work and presentations that are clear, relevant, concise, well- organized, and appropriate to the audience. OVERSIGHT ACTIVITIES HAD NOT SIGNIFICANTLY IMPROVED Although we criticized Region 8 project oversight in our 1995 review, project managers did not significantly revise their oversight activities to ensure contractors complied with project plans and site history files supported decisions and justified resources. Project managers continued to rely too heavily on oversight contractors and six of seven had not appropriately documented their oversight activities. Project managers did not periodically observe contractor field activities and, therefore, could not independently confirm contractors complied with project plans. Also, E1SKG7-08-5001-8100082 28 ------- Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls some project managers did not maintain adequate site history files that would support site management decisions. Some project managers did not revise their oversight activities. Three of seven project managers continued to rely on their field sampling and oversight contractors to conduct sampling and provide oversight of other contractors and did not personally observe field sampling activities as directed in the Regional policy memorandum. Although project managers received contractor- or state- prepared progress reports to help them monitor site activities, they did not regularly schedule site visits to observe contractor performance or sampling activities. As such, project managers could not ensure contractors performed as expected in workplans, and project managers did not broaden their experience and knowledge about their sites' sampling activities. As we reported in 1995, inadequate oversight increased the risk that contractors might not be performing in accordance with statements of work and workplans. Also, contractors could have circumvented sampling requirements since project managers did not periodically observe contractors' field activities. Six of seven project managers had not appropriately documented their oversight actions and maintained a complete site history file to support field sampling and other decisions, resources expended, and site transitions between project managers or between EPA and states. One project manager voiced concern about assuming responsibility for a site that had an incomplete site history file because a previous project manager did not prepare memorandums-of- record to document what had transpired at the site. The project manager explained that file documentation should provide evidence that supports: (1) what actions occurred, (2) what agreements were reached, (3) how funds were spent, and (4) how resources were used when transitioning a site from one project manager to another or when transitioning a site from EPA to the state. Without documentation that supported oversight actions, newly assigned project managers were less efficient and effective. E1SKG7-08-5001-8100082 29 ------- Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls OVERSIGHT RESPONSIBILITIES AND AUTHORITY WERE UNCLEAR Project managers had no procedures clearly describing their oversight and site management responsibilities and authority. The Regional policy memorandum was too general, and did not provide specific oversight procedures that directed project managers' actions and site management activities. Because some project managers did not believe they had specific procedures, they stated they performed their project manager responsibilities in accordance with other directives, such as consent decrees. They explained that consent decrees provide specific requirements or actions, such as defining documentation requirements and stipulating timelines for construction activities. Project managers explained that they wanted to maintain their authority to apply their professional judgment when making site decisions but would welcome better direction. They explained that part of their reliance on contractors resulted from competing priorities and insufficient time to complete all their work. They also explained that they knew they had authorities to enforce contractor compliance with workplans but were unsure whether they would get management support if they tried to take action against a contractor for poor performance. Some project managers explained that they used a tiered approach when managing contractors. They stated that if they had extensively used a contractor and had established confidence in the contractor's abilities, the project managers would allow the contractor more discretion and not exercise as much oversight as would be necessary if the contractor was inexperienced or new to the site. Program supervisors did not ensure staff properly executed oversight responsibilities. One supervisor stated that he allowed his staff to use their professional judgment as to site visits and what information needed to be documented. He believed his staff was fulfilling his expectations. Another supervisor explained that he counseled staff about documentation requirements but did not believe in micro- managing staff. Both supervisors believed their staff was appropriately performing oversight. E1SKG7-08-5001-8100082 30 ------- Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls PROJECT MANAGERS DID NOT VALUE QUALITY ASSURANCE ASSISTANCE Project managers did not value the quality assurance group's assistance and believed that the quality assurance group was unresponsive to their needs because the group's comments on project plans were not always constructive. Also, they believed the quality assurance group's implementation of quality assurance rules was often too rigid. They stated the group needed to be more customer- oriented. As a result, project managers did not routinely solicit the group's input on project plans or consistently use the Region's sample brokering system. The Regional quality assurance program manager acknowledged that the quality assurance group had a negative image. Some project managers did not consistently use the quality assurance group to assist them during their reviews because they did not consider quality assurance comments to be constructive or value-added. Project managers stated that some quality assurance staff did not understand project managers' needs. Project managers also said they believed quality assurance staff required them to include information in plans that was not relevant to the particular site. However, some project managers provided very positive feedback about one quality assurance staff member who had a strong project manager background. On the strength of her background, this quality assurance staff member understood project managers' needs and provided timely, effective feedback. For example, she explained "line by line" any revisions to project plans that she believed were necessary. According to one project manager, when this staff member left the quality assurance group, subsequent quality assurance comments on project managers' plans did not provide the same specificity. Subsequent quality assurance comments were sometimes generic and required certain formats be followed before the quality assurance group would agree with the project plan contents. Asa result, project managers did not readily accept subsequent quality assurance comments and believed quality assurance requirements were too rigid. E1SKG7-08-5001-8100082 31 ------- Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls Project managers explained, and the quality assurance group agreed, that the sample brokering system was sometimes rigid and not user-friendly. For example, one project manager experienced problems scheduling analytical services. Regional policy required written requests for analytical services. However, the sample broker verbally authorized the project manager's request, but the Region 8 laboratory had not received the authorization and would not analyze her samples without a written authorization. The project manager had to obtain funding to use a commercial laboratory because the Region 8 laboratory would not analyze the samples on such short notice. The perception left with the project manager was that the requirement for a written authorization was too rigid. The project manager stated that other project managers also resist having to use a rigid sample brokering system or being told they must use this system. The quality assurance group acknowledged some scheduling problems and was aware that it needed to be more flexible. The sample broker and the quality assurance program manager have met with project managers to discuss how to improve the process. The Regional quality assurance program manager recognized that the quality assurance group's image was negative. He explained that the group had been working closer with program offices to change project managers' perceptions and help meet their program goals. Because the quality assurance group can be instrumental in helping project managers effectively perform their jobs, the quality assurance group must ensure its staff effectively communicate to promote quality assurance practices. CONCLUSION Regional project managers did not properly implement Region 8's controls or improve their oversight activities. Project managers had not significantly changed their oversight practices to ensure compliance with project plans. When project managers rely too heavily on contractors and E1SKG7-08-5001-8100082 32 ------- Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls do not observe contractor activities, they cannot ensure contractors performed as intended, be fully knowledgeable of site activities, or enhance their own experience. Without appropriate and documented oversight, Region 8 cannot ensure data is reliable and defensible. Controls may need to be reexamined to provide for improved contractor oversight and more responsible project manager actions. RECOMMENDATIONS We recommend the Regional Administrator: 5-1. Hold project managers accountable for following existing Region 8 policy and determine whether additional oversight direction is needed. 5-2. Schedule and budget for a minimum number of site visits to observe contractors' field sampling activities and oversight of other contractors. 5-3. Specify the type of site file documentation (e.g., agreements reached) required in site files to fully document field sampling decisions. 5-4. Assure staff that management will support staff decisions to take appropriate action if contractors perform poorly. 5-5. Determine whether project managers should be required to obtain Regional quality assurance assistance before approving a project plan. 5-6. Provide sample brokering standard operating procedures to project managers. E1SKG7-08-5001-8100082 33 ------- Region 8 Needs to Further Strengthen Its Superfund Field Sampling Oualitv Assurance Controls EXHIBIT 1 THE DATA QUALITY OBJECTIVES PROCESS COMPARED TO THE SCIENTIFIC METHOD The scientific method and the data quality objectives process provide tools to plan activities. EPA's quality system is based on the scientific method which consists of three stages: planning, implementation, and assessment. The 7-step data quality objectives process is comparable to the planning stage of the scientific method (steps 1-4). We compared the planning stage of the scientific method to the data quality objectives process. The following tables correlate the steps of the scientific method with those of the data quality objectives process. SCIENTIFIC METHOD 1. DEFINE A PROBLEM. 2. GATHER PERTINENT HISTORICAL/BACKGROUND INFORMATION. 3. FORM A WORKING HYPOTHESIS OR EXPLANATION. 4. DO EXPERIMENTS TO TEST THE HYPOTHESIS. 5. INTERPRET THE RESULTS. 6. DRAW A CONCLUSION AND MODIFY THE HYPOTHESIS. DATA QUALITY OBJECTIVES PROCESS 1. STATE THE PROBLEM. 2. IDENTIFY THE DECISION. 3. IDENTIFY INPUTS TO THE DECISION. 4. DEFINE THE STUDY BOUNDARIES. 5. DEVELOP A DECISION RULE. 6. SPECIFY LIMITS ON DECISION ERRORS. 7. OPTIMIZE THE DESIGN FOR OBTAINING DATA. Scientific Method Data Quality Objectives STEPS 1 1,2 2 3 3 5,6 4 *4,7 5 — 6 — * Data quality objectives steps 4 and 7 should be continually assessed throughout a project. E1SKG7-08-5001-8100082 34 ------- Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls EXHIBIT 2 SCOPE AND METHODOLOGY We conducted our fieldwork from January 1997 through November 1997 in accordance with Government Auditing Standards (1994 Revision) issued by the Comptroller General of the United States. We designed our followup audit to gather information that would determine whether the Region's corrective actions improved its field sampling quality assurance activities after issuance of our January 1995 report. We judgmentally selected 10 (approximately 25 percent) Region 8 Superfund sites from EPA Region 8's National Priority JJst. Region 8's list included 5 states with a total of 38 general Superfund sites and 9 federal facilities sites. We did not include the federal facilities sites within our universe. We judgmentally selected two sites from each of the five states except South Dakota and Colorado. South Dakota only listed two sites of which one was a federal facility site. In place of the South Dakota site, we selected a third site from Colorado, as Colorado had the majority of sites listed. We judgmentally selected the sites based on size, money spent, and time listed on the National Priority List. We judgmentally selected 36 project plans for these sites that included sampling and analysis plans, quality assurance project plans, and field sampling plans. We selected the plans based on site index sheets provided by the Superfund Records Center. We evaluated the project plans to determine whether Region 8 adequately implemented and documented its data quality objectives process. We reviewed Regional policy memorandums issued in response to our 1995 report and examined the same 36 project plans to determine whether project managers had complied with the Region's additional controls. Because we were interested hi Regional improvements after our 1995 report, we narrowed those plans to 8 that the Region developed and implemented after our 1995 report. The additional Regional controls included having signed signature pages within the project plans and requiring all analytical service requests to go through the Region's sample brokering system. Also, we interviewed 7 project managers that managed 8 of the 10 sites hi our review and 2 Superfund remedial program supervisors about their awareness of the additional controls and staffs' implementation of the Region's revised policies. We interviewed the same seven project managers and supervisors about their mandatory quality assurance training attendance and understanding of quality assurance concepts, especially related to the data quality objectives process. We relied on training records from the regional quality assurance program manager, the remedial program training coordinator, and the Montana training coordinator to determine whether the appropriate project managers and other remedial staff had attended training. Region 8 changed its training requirements from its 1992 to 1996 Quality E1SKG7-08-5001-8100082 35 ------- Management Plans. We used the Region's 1996 Quality Management Plan to assess staffs compliance with training requirements. We reviewed Regional policy memorandums issued in response to our 1995 report. We interviewed the same seven project managers about their oversight responsibilities, available guidance, and implementation of revised Regional policy memorandums to determine whether project managers had changed their past oversight practices in accordance with the revised Region 8 policy. E1SKG7-08-5001-8100082 36 ------- Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls EXHIBIT 3 EPA COMMENTS AND OIG EVALUATION Region 8 generally concurred with our recommendations and presented an action plan that outlined how the Region planned to address our recommendations. The action plan generally satisfied the intent of our recommendations. We agree that Region 8 had improved its field sampling quality assurance controls since our last report. We also agree that the Region can further improve its quality assurance efforts in its Superfund field sampling planning. We reviewed Region 8's specific comments to our draft report and made some minor modifications to the final report as we determined necessary. Region 8 made some comments for which we made no changes to the report. We did not revise our final report to include Region 8 comments about the basin-wide quality assurance project plan because we still conclude that the site-specific sampling and analysis plan needed specific data quality objectives. We also did not change our position on the sample brokering system. We agree that the Region cannot direct contractors to select a specific subcontractor for laboratory analysis. However, the Region may direct contractors to use the Region 8 laboratory for analysis which would require use of the sample brokering system. Furthermore, Regional policy requires project managers to notify the sample broker before any sampling is done and requires project managers to provide the sample broker with an approved sampling plan regardless of who is doing the laboratory analysis. We also did not revise our discussion of the untimely Summitville sampling and analysis plan because we believe it exemplifies the issue of timely plan review. We also did not change our position on inadequate oversight because inappropriate levels of oversight have persisted since our last audit. See Appendix I for the full Agency response. E1SKG7-08-5001-8100082 37 ------- UNITED STATES ENVIRONMENTAL PROTECTION AGENCY REGION VIII 999 18th STREET - SUITE 500 DENVER, COLORADO 80202-2466 APPENDIX I Page 1 of 5 FEB 1 3 1998 Ref: 8TMS-G MEMORANDUM SUBJECT: FROM: TO: Comments on Draft Report, Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality ssurance Controls,Audit Report E1SKG7-08-5001-XXXXXXX elldtotai Regional Administrator 1 Bennie S. Salem Divisional Inspector General For Audit Central Audit Division As requested in the December 23, 1997, cover memorandum to the above-referenced report, we are providing comments on the findings of the report, a concurrence or conditional concurrence with each of the recommendations presented, and an action plan that addresses each of the recommendations. We agree that further improvements can be made to quality assurance efforts in our Superfund field sampling planning. However, we also believe that significant improvements have been made since the previous Office of Inspector General (OIG) report issued in January 1995 and that project managers are implementing the program in a sound and responsible manner. We believe the draft report unfavorably describes our quality assurance program, drawing overall conclusions based on very narrow instances where we have fallen short. We suggest that more recognition be made of the actions taken in response to the previous report. We do appreciate the suggestions made by the OIG on how improvements could be made in our current processes and program controls. Despite our concerns with the overall project approach and the manner in which the findings are presented, we are planning to address the recommendations made, with the conditions listed below. An action plan is attached that provides information on how we plan to address those recommendations. 38 Printed on Recycled Paper ------- APPENDIX I Page 2 of 5 Our response to the report recommendations is as follows: 2-1 Concur, provided that the last sentence of the recommendation is deleted. See Action Plan. ,( 2-2 Concur. See Action Plan. 2-3 Concur. See Action Plan. 2-4 Concur. See Action Plan. 3-1 Concur. See Action Plan. 3-2 Concur. See Action Plan. 3-3 Concur, provided that the recommendation is modified to state something such as, "When a pattern of delay becomes evident, implement existing policies regarding late deliverables at a minimum." See Action Plan. 3-4 We understand that this recommendation is being deleted and will not appear in the final report. 3-5 Concur. See Action Plan. 4-1 Concur. See Action Plan. 4-2 Concur. See Action Plan. 4-3 Concur. See Action Plan. 4-4 Concur. See Action Plan. 4-5 Concur. See Action Plan. 5-1 Concur. See Action Plan. 5-2 Concur. See Action Plan. 5-3 Concur, provided that the phrase, "and other site decisions," is deleted. 5-4 Concur. See Action Plan. 5-5 Concur. See Action Plan. 5-6 Concur. See Action Plan. Please note that specific comments on the report are listed in Attachment 1. The Action Plan is described in Attachment 2. We would like to thank you for the opportunity to provide comments on the draft report. For questions, please call Barbara Rodriguez, Regional Audit Coordinator, at (303) 312-6360. 39 Printed on Recycled Papar ------- APPENDIX I Page 3 of 5 Attachment 1 Specific Comments on Report Chapter 2: Throughout this chapter, it should be made clear that project managers have had the responsibility for approval of quality^assurance documents only since 1992. Prior to that time, the Regional Quality Assurance Officer had the responsibility. For example, the first sentence on p. 7 should be clarified. p. 7, last paragraph. The Quality Assurance Project Plan developed for the entire Clark Fork Basin was approved on May 20, 1992. It addressed DQO's in an adequate manner with the understanding that each subsequent SAP would include any site- specific DQO's as appropriate. Much of the design work included in the Streamside Tailings Operable Unit was to further characterize the area as an extension of the RI/FS, and thus the decision was made that the basin-wide QAPP should be the governing plan. The State of Montana is now the lead for construction of the remedy. DQO's for that aspect of the remedy should be part of the planning process for the state's work under a cooperative agreement. p. 11, 2nd paragraph. The sample brokering program was designed only for "analytical work to be performed for all EPA Region VIII programs ... through lAGs, States, and contractors." Consequently it does not apply to PRP-lead sites. Also, the brokering system does not apply to those instances where laboratory services were being subcontracted out. That understanding is reflected in the Regional Quality Management Plan. The brokering system thus applies only to situations where EPA itself or EPA through an IAG or contract is directly responsible for obtaining analytical services through the Regional Laboratory, the Contract Laboratory Program, or other vehicle. p. 11, 2nd paragraph. The sample broker does not review each sampling plan. He only knows that the plan is approved. He then receives- a copy of the analytes. P. 12, 1st full paragraph. This example of the Summitville sampling plan does not demonstrate a situation that would be helped by requiring a 30-day review period. The actual situation is that the contractor was tasked with preparing a sampling plan. Generally these plans are prepared with significant data user input. However, in this case the EPA data user was not able to spend the time necessary to allow preparation of an adequate document. Thus when the draft plan came in, there had been a project management change that resulted in a new data user 40 ------- APPENDIX I Page 4 of 5 2 becoming involved. This new person decided that the plan did not meet EPA needs and asked that another plan be prepared. It was estimated that the costs incurred to date were $28,000. We do not believe that a 30-day requirement on a draft plan could have helped a situation that was directly related to the data user needs changing toward the end of the planning effort. p. 12, 2nd paragraph. First sentence should begin, "Some project managers," rather than the blanket statement, "Project managers" which makes the reader think that all or a majority of, project managers were not diligent in carrying out their duties. p. 12, 2nd paragraph. We have searched our records and have found approval letters for two of the five plans that did not have approval signatures. Approval documentation for these is attached as Attachments 1-A and 1-B. A third plan, for Butte Priority Soils, was approved based on the findings in the OIG draft report (Attachment 1-C). These approval letters were not attached to the sampling plans. One additional plan, for Streamside Tailings, was never implemented because EPA took back the work and gave it to the State. Therefore, the State would have the responsibility, for approving the plan. Thus, there is only one plan that has not been approved. -> p. 13, top of page. Region 8 already has in place mechanisms for holding PRPs and contractors accountable. PRP work is done under orders or decrees that specify the work to be done. In agreements there are stipulated penalties for late submission of deliverables. In unilateral orders, there is the threat of enforcement. Penalties are determined by a number of factors and are at the discretion of the project coordinator and signatory. For contracts the contract document itself describes the procedure to be followed in cases of lack of performance. Work assignments are evaluated and awards can be reduced by the Performance Evaluation Board based on the criteria in the contract. p. 16, 3rd paragraph. Attachment 1-D is a table that summarizes our findings on QA training needs. We believe that seven project managers, including a new-hire, currently require at least some training. As of September 30, 1997, we believe there were eight project managers needing training. One person, Ken Wangerud, who was identified in the report as needing training, has been exempted from some training according to the Rick Edmonds transition plan memo of January 9, 1996. Another, Lisa Lloyd, completed her training after September 30. Those who are currently untrained have been counseled about not signing plans until they are fully trained. We agree that the two SEE employees should be trained. Although not required by the Regional QAMP, we agree that some familiarity with QA requirements is important for managers of project managers. The 41 ------- APPENDIX I Page 5 of 5 QA Program will be developing a short course specific to this group. p. 18. first sentence as continued from p. 17. We disagree with the statement that managers are delivering a message that QA is not important, using the example of not responding to the request for information. Without supporting information, it is presumptuous of the OIG to conclude that the reason for not responding is a view that QA is not important. Even though there may be a perception of this view by some people in the QA Office, in fact, a memo was distributed to all RPMs in the Superfund Remedial Response Program from the Program Director on January 16, 1996, that clearly states the importance of receiving this training. This was one week after the transition plan memo was issued. p. 20, second paragraph under "Oversight..." It is not clear how widespread this problem is. We believe there may be a few project managers who may not have been present at their sites during a field sampling episode. However, project managers routinely perform site visits for one purpose or another and personally, or with state staff assistance, oversee contractor or PRP work. Site-specific travel is routinely approved and encouraged as needed to assure proper oversight. p. 20, third paragraph under "Oversight..." Same as above. 42 ------- Region 8 Needs to Further Strengthen Its Superfund Field Sampling Quality Assurance Controls APPENDIX II DISTRIBUTION Office of Inspector General Inspector General (2410) Deputy Assistant Inspector General for Internal Audits (2421) Headquarters Audit Liaison (2421) Divisional Inspectors General for Audit Headquarters Office Assistant Administrator for Solid Waste and Emergency Response (5101) Director, Office of Emergency and Remedial Response (5201) Division Director, Analytical Operations and Data Quality Center (5202G) Division Director, Regions 5/7 Accelerated Response Center (5202G) Assistant Administrator for Research and Development (8101) Director, National Center for Environmental Research and Quality Assurance (8701) Division Director, Quality Assurance Division (8724R) Agency Followup Official (2710) Agency Followup Coordinator (3304) Associate Administrator for Regional Operations and State/Local Relations (1501) EPA Region 8 Office Assistant Regional Administrator for Ecosystems Protection and Remediation (8EPR) Assistant Regional Administrator for Technical and Management Services (8TMS) Director, Montana Operations Office (8MO) Program Manager, Superfund Remedial Response - Denver (8EPR-SR]. Program Manager, Superfund Remedial Response - Montana (8MO) Program Manager, Quality Assurance (8TMS-Q) Audit Coordinator (8TMS-G) E1SKG7-08-5001-8100082 43 ------- |