Office of Inspector General
       Audit Report
            SUPERFUND
     Region 8 Needs to Further Strengthen Its
Superfund Field Sampling Quality Assurance Controls
        Report No. E1SKG7-08-5001-8100082
               March 25,1998

-------
Inspector General Division          Central Audit Division
 Conducting the Audit:             Denver, Colorado
Region Covered:                  Region 8
Program Office Involved:           Office of Ecosystems Protection
                                  And Remediation

                                 Office of Technical and Management
                                  Services

-------
                UNITED STATES ENVIRONMENTAL PROTECTION AGENCY

                                 OFFICE OF INSPECTOR GENERAL
    _    :                          CENTRAL AUDIT DIVISION

V^V                         J&WgfiXX*                       B-CH0mCES
  *i PBO^                                                                    1445 ROSS AXENJS. SUITE 1:0;
                                          913-551-7878                         DALLAS TEXAS 75202-:-;:-
                                        FAX: 913-551-7837                                 2i--e-:s-eei'
                                                                                     FAX 21--665-6589
                                                                              999 18Tn STPEE" SU TE 50C
                                                                         DENVER C 0,0= ADC 80202-2-:5
                                           March 25, 1998                            FAX go^lsS
    MEMORANDUM

    SUBJECT:    Region 8 Needs to Further Strengthen Its Superfund
                   Field Sampling Quality Assurance Controls
                  Audit Report E1SKG7-08-5001-8100082

    FROM:       Bennie S. Salem   &^~it^ v^J-£^~—.
                  Divisional Inspector General

    TO:           William P. Yellowtail
                  Regional Administrator
                  Region 8

           We have completed our followup audit of the U. S. Environmental Protection Agency (EPA)
    Region 8's Superfund field sampling quality assurance activities.  Attached is our report entitled Region
    8 Needs to Further Strengthen Its Superfund Field Sampling Quality- Assurance Controls. We found
    that the Region needed to better define and document its data quality objectives, further strengthen
    controls over field sampling activities, improve its quality assurance training to its remedial project
    managers, and ensure increased oversight and improve documentation of oversight at Superfund sites.
    We met with the Region 8 Superfund Remedial Response program manager and supervisor and the
    Regional Quality Assurance program manager who generally concurred with our recommendations and
    agreed to begin corrective actions. We incorporated their comments as appropriate.

    Action Required

          In accordance with EPA Order 2750, you, as the action official, are required to provide us a
    written response to the audit report within 90 days of the final audit report date.  For corrective actions
    planned but not completed by the response date, reference to specific milestone dates will assist us in
    deciding whether to close this report.

          This audit report contains findings that describe issues the Office of Inspector General (OIG)
    has identified and corrective actions OIG recommends.  This report represents the opinion of OIG, and
    the findings contained in this report do not necessarily represent the final EPA position. Final
    determinations on matters in this report will be made by EPA managers in accordance with established
    EPA audit resolution procedures.  In this audit, OIG did not measure the reviewed offices against all
    standards established by the National Contingency Plan.  The findings contained in the report are not
    binding in any enforcement proceeding brought by EPA or the Department of Justice under Section

                                                                              RECYCLED

-------
107 of the Comprehensive Environment Response, Compensation, and Liability Act to recover costs
incurred not inconsistent with the National Contingency Plan.

       We have no objections to the release of this report to any member of the public. This report
contains no confidential business or proprietary information that cannot be released to the public.

       Should you or your staff have any questions regarding this report, please contact me at (913)
551-7831 or Jeff Hart, Audit Manager in our Denver office, at 312-6169.  Please refer to the report
number on all related correspondence.

Attachment

-------
                                                Region 8 Needs to Further Strengthen Its Superfund
                                                       Field Sampling Quality Assurance Controls
                         EXECUTIVE SUMMARY
INTRODUCTION
 The Office of Inspector General conducted a followup audit
 of Region 8's Superfund field sampling quality assurance
 activities. In our January 1995 report, Implementing
 Controls Would Improve Region 8's Superfund Field
 Sampling Activities (E1SKG4-08-0045-5400034), we
 found that Region 8 needed to strengthen pre-field sampling
 controls, better train its remedial project managers (project
 managers), and ensure an appropriate level of oversight at
 Superfund sites. Our followup audit disclosed that similar
 problems continued to exist.

 The U. S. Environmental Protection Agency (EPA) requires
 that each region or major organization have a quality
 assurance program to ensure that activities generate data of
 known and adequate quality. EPA stated in its 1997
 strategic plan that it "will continue to implement its
 mandatory Quality Assurance Program" to ensure high
 quality environmental data.
OBJECTIVES
Our general objective was to review Region 8's improved
procedures and processes for review, approval, and
implementation of field sampling quality assurance activities
in the Superfund Remedial Program. Specifically, our
objectives were to answer the following questions:

       *     Is the Region properly using data quality
             objectives to support decisionmaking?

       *     Are adequate controls in place to ensure that
             an approved sampling and analysis plan is in
             place prior to commencing field sampling
             activities?
                                                                 E1SKG7-08-5001-8100082

-------
                                              Region 8 Needs to Further Strengthen Its Superfund
                                              	Field Sampling Quality Assurance Controls
                                               Do project; managers possess the skills
                                               necessary, through training, to approve
                                               sampling and analysis plans?

                                               Is Regional oversight of field sampling
                                               activities adequate to ensure compliance with
                                               sampling and analysis plans and reliable and
                                               defensible data?
RESULTS IN BRIEF
A Consistently Applied Data
Quality Objectives
Process Would Provide For
Better Decisions
Region 8 improved some of its field sampling quality
assurance activities since 1995. However, the Region could
more effectively support EPA's goal for high quality
environmental data by better implementing its quality
assurance program. Project managers could better define,
document, and update data quality objectives to support
environmental cleanup decisions. Also, Region 8's
strengthened controls did not fully achieve their intended
purposes to prevent untimely project plan submissions and
undocumented plan approvals. Supervisors had not ensured
that all parties responsible for field sampling activities
attended quality assurance training. Project managers
continued to perform limited oversight over contractors and
did not routinely look to the Region 8 quality assurance
group for assistance in reviewing project plans.

Project managers could have more consistently applied the
data quality objectives process as specified by EPA policy.
Specifically, they should have better defined, documented,
and updated data quality objectives to support their
environmental cleanup decisions. Data quality objectives  in
EPA project plans were too broadly defined to ensure
sufficient and appropriate data was gathered, and project
managers accepted deficient contractor-prepared plans.
Without evidence of decisions based on sound data, EPA is
vulnerable to criticism and risks inadequate, inappropriate, or
untimely sampling and cleanup.
                                                                    E1SKG7-08-5001-8100082

-------
                                               Region 8 Needs to Further Strengthen Its Supeifund
                                              	Field Sampling Quality Assurance Controls
Accountability For
Strengthened Controls
Would Help Eliminate
Planning Problems
Training Would Strengthen
Quality Assurance Skills
Further Actions Could Ensure
Appropriate
And Well-Documented
Oversight
Region 8's strengthened controls did not fully achieve their
intended purposes to prevent untimely project plan
submissions and undocumented plan approvals.  Some
project managers continued to allow contractors and
potentially responsible parties to submit late plans and
neglected to ensure their plans included signed approval
pages. Without clearly identified and approved plans, the
Region could not ensure it (1) collected the necessary data to
accurately identify pollution sources and assess risks, or (2)
made the most efficient, effective, or scientifically defensible
cleanup decisions.

Region 8 agreed that project managers needed to attend
mandatory quality assurance training to ensure they had the
necessary skills to properly approve project plans. However,
13 (or 38 percent) of the project managers and other staff
needing the training had not completed the two mandatory
Regional courses.  Management should have held project
managers accountable for attending training, and the
Regional quality assurance group needed to improve training
effectiveness.  Without well-trained project managers,
Region 8 could not ensure project managers had the
necessary skills to properly review and approve project
plans.

Region 8 had inadequate assurance that data collected during
the remedial cleanup phase was reliable and defensible
because project managers had not significantly changed their
oversight practices to ensure compliance with project plans.
Some project managers continued to perform limited
oversight, relied too heavily on oversight contractors, did not
use the Region 8 quality assurance group's assistance, and
had not appropriately documented their oversight activities.
Project managers had no procedures specifically describing
their oversight and site management responsibilities and
authority.  Without appropriate and documented oversight,
Region 8 could not ensure data was reliable and defensible.
                                                                    E1SKG7-08-5001-8100082
                                            in

-------
                                             Region 8 Needs to Further Strengthen Its Superfund
                                            	Field Sampling Quality Assurance Controls
RECOMMENDATIONS
We recommend that the Regional Administrator (RA)
develop Regional controls to ensure that Regional staff
appropriately develop data quality objectives^ document the
process in the project plans, and hold Superfund supervisors
accountable for ensuring that project managers have
actually reviewed and properly approved project plans.
Further, the RA should direct supervisors to identify staff
who have not completed required training and hold staff
accountable for attending. The RA should also direct
supervisors to hold project managers accountable for
following existing Region 8 policy and determine whether
additional oversight direction is needed.
AGENCY COMMENTS
AND OIG EVALUATION
Region 8 generally concurred with our recommendations
and presented an action plan that outlined how the Region
planned to address our recommendations and generally
satisfied the intent of our recommendations. We agree that
Region 8 had improved its field sampling quality assurance
controls since our last report. We also agree that the
Region can further improve its quality assurance efforts in
its Superfund field sampling planning. We reviewed Region
8's specific comments to our draft report and made some
minor modifications to the final report as we determined
necessary.
                                                                  E1SKG7-08-5001-8100082
                                           IV

-------
                                      Region 8 Needs to Further Strengthen Its Super-fund
                                            Field Sampling Quality Assurance Controls
                      TABLE OF CONTENTS

                                                                      Page

EXECUTIVE SUMMARY	  i
ABBREVIATIONS 	  vii

CHAPTERS

1     INTRODUCTION	1
            Purpose	1
            Background 	1
        ^   Scope And Methodology	4
            Prior Audit Coverage	4

2     A CONSISTENTLY APPLIED DATA QUALITY OBJECTIVES
       PROCESS WOULD PROVIDE FOR BETTER DECISIONS	6
            EPA Specifies Using The Data Quality Objectives Process 	6
            Managers Needed to Demonstrate Use of The Data Quality
             Objectives Process  	8
            Project Managers Accepted Deficient Plans  	11
            Conclusion	11
            Recommendations 	12

3     ACCOUNTABILITY FOR STRENGTHENED CONTROLS WOULD
       HELP ELIMINATE PLANNING PROBLEMS	13
            EPA Requires Timely Reviewed and Approved Plans 	13
            Project Managers Did Not Properly Apply Additional
             Controls	14
            Accountability Had Not Been Enforced	18
            Conclusion	19
            Recommendations	19

4     TRAINING WOULD STRENGTHEN QUALITY ASSURANCE SKILLS	21
            EPA and Region 8 Require Quality Assurance Training	21
            Staff Had Not Completed Required Training 	22
            Management Did Not Regularly Emphasize Quality Assurance Training 	23
            Project Managers Did Not Value Quality Assurance Training	24


                                                        E1SKG7-08-5001-8100082

-------
                                      Region 8 Needs to Further Strengthen Its Superfund
                                      	Field Sampling Quality Assurance Controls
            Conclusion	25
            Recommendations 	26
5     FURTHER ACTIONS COULD ENSURE APPROPRIATE AND
       WELL-DOCUMENTED OVERSIGHT	27
            Project Managers' Responsibilities Include Oversight 	27
            Oversight Activities Had Not Significantly Improved	28
            Oversight Responsibilities and Authority Were Unclear	30
            Project Managers Did Not Value Quality Assurance
             Assistance	31
            Conclusion	32
            Recommendations  	33
EXHIBITS

1     THE DATA QUALITY OBJECTIVES PROCESS COMPARED TO
       THE SCIENTIFIC METHOD 	34

2     SCOPE AND METHODOLOGY	35

3     EPA COMMENTS AND OIG EVALUATION  	37


APPENDICES

I     EPA RESPONSE	38

H     DISTRIBUTION	43
                                                        E1SKG7-08-5001-8100082
                                    vi

-------
                                   Region 8 Needs to Further Strengthen Its Superfund
                                   	Field Sampling Quality Assurance Controls
                      ABBREVIATIONS
EPA               U. S. Environmental Protection Agency

OIG               Office of Inspector General

Project Manager     Remedial Project Manager
                                                       E1SKG7-08-5001-8100082
                                 vii

-------
                                            Region 8 Needs to Further Strengthen Its Superfund
                                            	Field Sampling Quality Assurance Controls
                                  CHAPTER 1
                                 INTRODUCTION
PURPOSE
The Office of Inspector General (OIG) conducted a
followup audit of Region 8's Superfund field sampling
quality assurance activities.  Our general objective was to
review Region 8's improved procedures and processes for
review, approval, and implementation of field sampling
quality assurance activities in the Superfund Remedial
Program. Specifically, our objectives were to answer the
following questions:

       *•      Is the Region properly using data quality
             objectives to support decisionmaking?

       *•      Are adequate controls in place to ensure that
             an approved sampling and analysis plan is in
             place prior to commencing field sampling
             activities?

       *•      Do project managers possess the skills
             necessary, through training, to approve
             sampling and analysis plans?

       *      Is Regional oversight of field sampling
             activities adequate to ensure compliance with
             sampling and analysis plans and reliable,
             defensible data?
BACKGROUND
The U. S. Environmental Protection Agency (EPA)
requires that each region or major organization have a
quality assurance program to ensure that activities generate
data of known and adequate quality.  EPA stated in its 1997
strategic plan that it "will continue to implement its
mandatory Quality Assurance Program" to ensure high
                                                                 E1SKG7-08-5001-8100082

-------
            Region 8 Needs to Further Strengthen Its Superfund
                   Field Sampling Quality Assurance Controls
quality environmental data. The quality assurance program
goal is to measure and minimize sources of error and to
monitor conditions of sampling, storage, and transport.

Quality assurance and quality control operate hand-in-hand
to help managers meet EPA's quality assurance goal.
Quality assurance is a management function that should
provide policy and an integrated system of management
controls for planning, implementation, and review of
environmental data collection activities.  Quality control is a
system designed to measure the attributes and performance
of an activity against specific standards and verify that they
meet the standards.  Quality control should be implemented
at the project level and include such scientific precautions as
calibrations and duplications.

EPA Order 5360.1, Policy and Program Requirements to
Implement the Mandatory Quality Assurance Program,
dated April 1984, applies to all EPA programs including
Superfund. It provides for each EPA organization to
implement the requirements of EPA's quality assurance
program.  EPA Order 5360.1 requires data be of known
quality and states that data are of known quality when data
"...are thoroughly documented, such documentation being
verifiable and defensible."

EPA developed a "data quality objectives" process as its
systematic planning tool to help ensure it consistently
collected the quality of data needed to support technical
decisions.  The data quality objectives process is based on
the scientific method and thus contributes to EPA's goal of
sound science.  EPA requires project plans to include a data
quality objectives section to document the project's defined
goal and planning outputs.

The National  Contingency Plan regulates the Superfund
Remedial Program through Part 300 of 40 Code of Federal
Regulations.  Sections 300.420 and 300.430 require
sampling and analysis plans that provide a process for

                                 E1SKG7-08-5001-8100082

-------
            Region 8 Needs to Further Strengthen Its Superfund
           	Field Sampling Quality Assurance Controls
obtaining environmental data of sufficient quality and
-quantity to support decisionmaking. EPA must review and
approve sampling and analysis plans at most Superfund
sites. Sampling and analysis plans consist of:  (1) a field
sampling plan, and (2) a quality assurance project plan.  A
field sampling plan describes the number, type, and location
of samples and the type of analysis required. The quality
assurance project plan describes policy, organization, and
functional activities necessary to  develop adequate data.
The data is used for planning and documenting a removal
action, site evaluation, and hazard ranking system activities.
EPA staff often use the terms "sampling and analysis plan,"
"field sampling plan," and "quality assurance project plan"
interchangeably. Throughout this report, we have used the
term "project plan" to refer to all these types of plans except
when we have described a specific example.

EPA's quality assurance program requires each organization
that collects environmental data to implement an approved
quality management plan.  Region 8's Quality Management
Plan, dated September 1996, is Region 8's statement of and
commitment to quality assurance activities. It describes
Region 8 quality management policies, objectives,
principles, responsibilities, and accountability.

Region 8's Office of Ecosystems Protection and
Remediation and Region 8's Montana Office are responsible
for Superfund cleanup and oversight.  On-scene
coordinators, site assessment managers, and remedial
project managers (project manager) are responsible for
ensuring field sampling integrity at their sites and ultimately
report to the Superfund program managers located in
Denver and Montana.

Region 8's Office of Technical and Management Services
has responsibility for the quality assurance program. The
quality assurance group oversees the quality of data-
generating activities throughout the Region.
                                 E1SKG7-08-5001-8100082

-------
                                             Region 8 Needs to Further Strengthen Its Super-fund
                                            	Field Sampling Quality Assurance Controls
SCOPE AND
METHODOLOGY
We performed a followup audit of EPA Region 8's
Superfund field sampling quality assurance activities to
determine whether Region 8 improved its controls over its
field sampling activities as a result of our report,
Implementing Controls Would Improve Region 8's
Superfund Field Sampling Activities, E1SKG4-08-0045-
5400034, dated January 27, 1995. We performed our audit
in accordance with Government Auditing Standards (1994
Revision) issued by the Comptroller General of the United
States. Other than the issues discussed in this report, no
other significant issues came to our attention that warranted
expanding the scope of our review. The scope of our 1995
report and our followup audit work was limited to the
Superfund Remedial Program. See Exhibit 2 for additional
information about our scope and  methodology.
PRIOR AUDIT
COVERAGE
The only relevant prior audit coverage was the report cited
above. We have summarized relevant portions of the report
below.

We found that project managers inconsistently approved
project plans and did not denote final, approved plans.
Project managers previously allowed contractors to submit
project plans for review and approval without providing 30
days for EPA review and approval prior to a sampling
event. At the time, Region 8 allowed verbal approvals.
Consequently, project managers had not documented their
review and approvals in project plans. Some project
managers prepared approval letters, but did not attach the
letters to the approved plan or identify the approved plan in
the letter.

Our 1995 report disclosed that project managers recognized
they had not fully complied with mandatory training
requirements within specified timeframes or understood
quality assurance requirements even after having attended
Regional quality assurance courses.  Project managers
                                                                 E1SKG7-08-5001-8100082

-------
            Region 8 Needs to Further Strengthen Its Superfund
           	Field Sampling Quality Assurance Controls
identified weaknesses in training and requested better
presentations to include identifying quality assurance
experts for project managers to contact. None of the
project managers had completed all three mandatory quality
assurance courses required by the Region as of January
1994.  Also, the Region needed to better document training
attendance.

We further found that project managers relied on
contractors to conduct sampling with limited oversight, did
not always adequately oversee activities to ensure proper
contractor performance, and did not enhance their own
experience by observing site activities.  Project managers
had not used field audits to help them oversee their sites.

In response to our 1995 report, the Region instituted
strengthened controls  over its field sampling activities by
distributing two internal policy memorandums. The first
memorandum to all Region 8 office directors instructed
them to use an existing Region 8 sampling brokering system
for all analytical work. Project managers were to attach
approved and signed project plans to their requests for
analytical services. Requests were required 1-6 weeks
before  a sampling event. The Region agreed that project
managers and contractors should have followed a 30-day
approval rule and suggested that staff include the
requirement in contractor project planning schedules. The
second memorandum  instructed project managers to
approve project plans  in writing prior to field sampling and
include the approval pages within the document. The
Region further directed its project managers to take the
mandatory quality assurance training. The memorandum
advised project managers of other related quality assurance
and program training available to enhance their skills. The
memorandum emphasized contractor oversight and
provided suggestions of where to get additional technical
expertise to review project plans, how to use field audits to
fulfill oversight responsibilities, and how to document site
decisions.
                                 E1SKG7-08-5001-8100082

-------
                                            Region 8 Needs to Further Strengthen Its Super-fund
                                           	Field Sampling Quality Assurance Controls
                                  CHAPTER 2

            A CONSISTENTLY APPLIED DATA QUALITY OBJECTIVES
              PROCESS WOULD PROVIDE FOR BETTER DECISIONS

                                  Project managers could have more consistently applied the
                                  data quality objectives process.  EPA policy and guidance
                                  specifies using the data quality objectives process to
                                  support site decisionmaking.  Project managers should
                                  have better defined, documented, and updated data quality
                                  objectives to support their environmental cleanup
                                  decisions. Region 8's project plans did not provide
                                  evidence that Region 8 used the data quality objectives
                                  process or another systematic planning approach to
                                  support decisions. Project managers stated they routinely
                                  reviewed contractors' project plans, but our review
                                  indicated they routinely accepted deficient plans. Without
                                  evidence of decisions based on sound data, EPA is
                                  vulnerable to criticism by Congress, states, potentially
                                  responsible parties, and the public for site mismanagement
                                  and unnecessary expenditures.   Additionally, it risks
                                  inadequate, inappropriate, or untimely sampling and
                                  cleanup.
EPA SPECIFIES USING
THE DATA QUALITY
OBJECTIVES PROCESS
The Deputy Administrator directed in a May 1984
memorandum, Data Quality Objectives, that "Beginning in
September 1984, data quality objectives should be an
integral part of each significant new data collection
activity." The Deputy Administrator further emphasized
that "each National Program Manager is responsible for
establishing data quality acceptance criteria (i.e., data
quality objectives) for all of their projects...."

A subsequent Deputy Administrator continued to focus on
the importance of data quality objectives in a November
1986 memorandum, Agency Institutionalization of Data
                                                                E1SKG7-08-5001-8100082

-------
           Region 8 Needs to Further Strengthen Its Superfund
           	Field Sampling Quality Assurance Controls
 Quality Objectives. This Deputy Administrator directed
 that EPA implement the data quality objectives process.
 The November 1986 memorandum stated that the data
 quality objectives process was "sound management
 planning" and a key element of EPA's quality assurance
 program. The memorandum also stated that data quality
 objectives are explicit statements of the quality of data
 needed to support a decision. The Deputy Administrator
 stated that institutionalization of this process would save
 money and improve data quality.

 General EPA Quality Assurance Division guidance and
 specific Superfund program guidance provide detailed
 instructions for project managers to implement the data
 quality objectives process to support decisionmaking. The
 Quality Assurance Division's September 1994 Guidance
for the Data Quality Objectives Process (QA/G-4) details
 a seven-step process to help project managers collect
 appropriate  data. The data quality objectives process is a
 systematic planning tool based on the scientific method for
 determining the type, quantity, and quality of data that will
 be sufficient and appropriate for the  data's intended use.
 Although project managers are not required to document
 each of the seven steps in project plans, the plans must
 adequately explain the results of the process. The Office  of
 Emergency  and Remedial Response also specifies using the
 process for all Superfund cleanups in its September 1993
 Data Quality Objectives Process for Superfund (EPA 540-
 R-93-071).  Region 8's Quality Management Plan requires
 project managers to develop and adequately document data
 quality objectives in their project plans that are appropriate
 for the site circumstances.
                                     w
 Because  the data quality objectives process is based on the
 scientific method, it "...improves the legal defensibility of
 site decisions by providing a complete record of the
 decision  process and criteria for arriving at conclusions."
 The scientific method provides principles and processes
 necessary for scientific investigation and a means for

                                E1SKG7-08-5001-8100082

-------
                                             Region 8 Needs to Further Strengthen Its Superfund
                                             	  Field Sampling Quality Assurance Controls
                                   independent observers to reach the same conclusion. See
                                   Exhibit 1 for a more detailed description of how the data
                                   quality objectives process builds upon the scientific
                                   method.

                                   Data quality objectives determine quality control
                                   requirements for data collection, sampling, and analysis.
                                   Data quality objectives provide the purpose and focus as
                                   well as the qualitative and quantitative criteria that:
                                   (1) define the most appropriate type of data to collect, and
                                   (2) specify the limits on decisionmaking errors to identify
                                   acceptable risk.  The seven steps in the data quality
                                   objectives process are shown in Exhibit 1.

                                   In accordance with the September 1994 data quality
                                   objectives guidance, data quality objectives must be
                                   continually assessed and revised as site circumstances
                                   change.  Also, the Quality Assurance Division's August
                                   1994 EPA Requirements for Quality Assurance Project
                                   Plans for Environmental Data Operations (QA/R-5)
                                   requires project managers to review project plans, at least
                                   annually, for long duration projects or multi-year projects.
                                   For example, Superfund sites sometimes take 10 years or
                                   more to clean up.  During such a lengthy cleanup process,
                                   much data is collected, many decisions are made, and
                                   public interests may change. In addition, new technology
                                   or new information on health risks may become available.
                                   Finally, site conditions may change as a result of
                                   unanticipated outcomes of other site decisions.
MANAGERS NEEDED TO
DEMONSTRATE USE OF
THE DATA QUALITY
OBJECTIVES PROCESS
Project managers did not use or adequately document the
use of the data quality objectives process in making field
sampling decisions. The data quality objectives for specific
projects did not adequately support decisionmaking
because project managers had not adequately: (1) defined
data quality objectives, (2) appropriately documented data
                                                                  E1SKG7-08-5001-8100082

-------
           Region 8 Needs to Further Strengthen Its Superfund
          	Field Sampling Quality Assurance Controls
quality objective results in their project plans, and (3)
updated project plans as site circumstances changed.

Although project managers stated they applied the data
quality objectives process, the objectives in their project
plans were too broadly defined to ensure sufficient and
appropriate data was gathered. Also, prior to 1992, the
quality assurance manager approved project plans but had
not identified deficient data quality objectives or ensured
the data quality objectives were well documented. We
examined 36 project plans and determined that the data
quality objectives in 8 plans (22 percent) were adequate,
while 28 (78 percent) were deficient.  The eight  adequate
plans were all prepared prior to the issuance of the 1994
data quality objectives guidance. These eight plans
appeared to follow the EPA data quality objectives process
and appropriately documented the specific project needs by
explaining the results of the planning process. The 28
deficient project plans included data quality objectives that
were broadly defined and were only task or project
objectives which did not directly contribute towards
specific site decisionmaking. While the majority of the 28
deficient plans were prepared prior to issuance of the 1994
guidance, 8 of the 28 were prepared after guidance
issuance.

As an example of the deficiencies, we noted the data
quality objectives for the California Gulch Superfund site
plan clearly stated a need to conduct a sampling study.
However, the objectives did not identify the point at which
enough information would be collected and a decision
could be made.  Appropriate data quality objectives for this
project should include statements of the problem (i.e., the
project/task objectives); specific inputs  needed; a decision
statement (i.e., a point at which remedial action should be
taken); risks associated with the decision, if the critical
sample results were too close to the action level; and an
optimal design.
                                 E1SKG7-08-5001-8100082

-------
           Region 8 Needs to Further Strengthen Its Super-fund
                   Field Sampling Quality Assurance Controls
Region 8 quality assurance staff told us that project plans
they reviewed over the last 2 years generally either did not
specify or incorrectly specified limits on acceptable
decision errors.  According to the Region 8 quality
assurance manager and the staff statistician, project
managers generally had not adequately identified limits on
decision errors (data quality objective step six).  Project
managers had not accurately calculated the total error and
could not have known the quality of the collected data.
While some plans identified an acceptable laboratory error
rate, they did not generally identify an acceptable field
error rate.  The plans generally implied that the laboratory
error rate was equal to the total error rate.  Therefore, the
plans did not demonstrate that implementation would result
in data of known quality.

Project managers did not update project plans to account
for changing site circumstances. All of the plans should
have contained a discussion of how the data quality
objectives evolved. However, only 1 of the 36 plans was
updated with this information. Plans for sites that
consisted of multiple operable units or phases referenced a
generic quality assurance project plan that was developed
many years before the most current sampling event. For
example, for the Silver Bow Creek/Butte, Montana site, a
1996 remedial design/remedial action sampling event
referred to the overall Site Remedial Investigation/
Feasibility Study Quality Assurance Project Plan dated
February 1992. This practice may have been considered
efficient for multiple remedial investigation/feasibility study
sample events. Using the 1992 project plan may also have
saved time and resources to prepare a new quality
assurance project plan for each event. However, multi-
year plans should be evaluated annually because the old
data quality objectives may no longer be relevant. In this
case, the 1996 sampling event was related to remedial
design/remedial action activities that are generally
significantly different from remedial investigation/feasibility
study objectives.

                                 E1SKG7-08-5001-8100082
         10

-------
                                              Region 8 Needs to Further Strengthen Its Superfund
                                             	Field Sampling Quality Assurance Controls
PROJECT MANAGERS
ACCEPTED DEFICIENT
PLANS
Although five of seven project managers stated that they
understood and applied the data quality objectives process,
the project plans we reviewed showed that project managers
did not understand what they were reviewing or chose to
accept plans with deficient data quality objectives sections.
Generally, project managers used contractors to prepare the
project plans we reviewed, including the data quality
objectives section. Project managers stated that they
provided the contractors with EPA guidance and
requirements documents to use when the contractor
prepared the project plans. Six of seven project managers
explained that they looked for data quality objectives
sections in their project plans during their reviews of the
plans. However, although contractors included the required
data quality objectives section in project plans, our review
of project plans showed that project managers had not
identified basic deficiencies in the data quality objectives
sections of the project plans or ensured that any data quality
objectives steps taken were adequately documented in the
project plans.

Some project managers did not understand that the data
quality objectives process could be modified to
accommodate site-specific needs. Some project managers
stated they believed the seven-step data quality objectives
process was unnecessary and inflexible. They explained that
not all seven data quality objective steps needed to be
completed for every data collection activity but did not
understand that not completing all steps was an acceptable
practice, if reasons for omitting steps were documented.
Instead, they viewed the process as inflexible and not
tailored to site-specific needs and phases of cleanup.
CONCLUSION
Region 8 unnecessarily risked inadequate or inappropriate
sampling and cleanup because it did not use or adequately
document application of the data quality objectives process.
Without adequate planning, EPA may waste resources
                                                                  E1SKG7-08-5001-8100082
                                           11

-------
                                              Region 8 Needs to Further Strengthen Its Superfund
                                             	Field Sampling Quality Assurance Controls
                                  developing sampling plans that are never used, collecting
                                  unnecessary samples, delaying cleanup, or cleaning up sites
                                  beyond what is cost effective.  While we believe project
                                  managers should complete all seven steps, the process is
                                  flexible and should be tailored to the significance of the
                                  sampling or other site decision to be made. The Region 8
                                  Quality Management Plan recognizes the process does not
                                  have to be complicated and laborious. Without proper
                                  documentation to support project managers' decisions, EPA
                                  is vulnerable to criticism by Congress and other outside
                                  parties. EPA must be able to clearly demonstrate that it has
                                  developed data of known quality, properly assessed risks,
                                  and that its decisions are based on the best scientific
                                  information available. To accomplish this, project managers
                                  must better define and document data quality objectives to
                                  adequately support their decisions.
RECOMMENDATIONS         We recommend the Regional Administrator:

                                  2-1.   Develop Regional controls to implement an effective
                                         data quality objectives process. Controls should be
                                         designed to ensure that Regional staff appropriately
                                         develop data quality objectives and adequately
                                         document the process in the project plans.

                                  2-2.   Direct project managers to review data quality
                                         objectives periodically throughout the cleanup
                                         process and replace or modify project plans to
                                         account for changing site circumstances.

                                  2-3.   Direct project managers to reject contractors'
                                         project plans that do not include an acceptable data
                                         quality objectives section.

                                  2-4.   Include in the controls for Recommendation 2-1,
                                         steps to ensure that contractor-prepared project
                                         plans include adequate data quality objectives.

                                                                   E1SKG7-08-5001-8100082
                                            12

-------
                                            Region 8 Needs to Further Strengthen Its Superfund
                                           	Field Sampling Quality Assurance Controls
                                 CHAPTER 3

              ACCOUNTABILITY FOR STRENGTHENED CONTROLS
                WOULD HELP ELIMINATE PLANNING PROBLEMS

                                 Region 8's strengthened controls did not fully achieve their
                                 intended purposes to prevent untimely project plan
                                 submissions and undocumented plan approvals.  EPA
                                 regulations and current Regional policy require timely and
                                 approved plans.  However, some project managers
                                 continued to allow contractors and potentially responsible
                                 parties to submit plans late without consequence, and some
                                 project managers neglected to ensure their plans included
                                 signed approval pages. Senior managers allowed staff to
                                 disregard improved controls developed in response to the
                                 1995 audit without being held accountable for results.
                                 Therefore, Region 8 could not be certain that project
                                 managers had adequate time to review plans or that proper
                                 sampling was done in accordance with approved plans.
                                 Without clearly identified and approved plans, the Region
                                 could not ensure it collected the necessary data to
                                 accurately identify pollution sources and assess risks. Also,
                                 the Region could not ensure it made the most efficient,
                                 effective, or scientifically defensible cleanup decisions.
EPA REQUIRES TIMELY
REVIEWED AND
APPROVED PLANS
EPA regulations and guidance and Region 8 policy require
approved project plans and records that are defensible and
verifiable. EPA Region 8 policy also specifies plans be
timely. The Superfund National Contingency Plan
regulations and Region 8 policy require that EPA review
and approve all project plans. EPA Order 5360.1 requires
written records that are defensible and verifiable. The Order
holds program managers responsible and requires that
information withstand reasonable challenges related to
accuracy or truthfulness and results that can be
substantiated. EPA Requirements for Quality Assurance
                                                                E1SKG7-08-5001-8100082
                                          13

-------
                                             Region 8 Needs to Further Strengthen Its Superfund
                                                    Field Sampling Quality Assurance Controls
                                 Project Plans for Environmental Data Operations (QA/R-
                                 5) states that plans must be approved prior to initiating site
                                 work.  Regional policy memorandums specified that project
                                 managers should ensure they receive project plans in
                                 advance of sampling activities to allow adequate time for
                                 thorough reviews. The policy memorandums instituted 30
                                 days as a standard.

                                 The Region  provides for accountability for fulfilling quality
                                 assurance requirements through performance agreements.
                                 Superfund remedial performance agreements for staff and
                                 supervisors require adherence to quality assurance policies,
                                 procedures,  guidelines, and regulations. Specifically,
                                 project managers must comply "...with data quality
                                 assurance policies, procedures, and guidelines to assure that
                                 data collected are of the quality and type needed to support
                                 decisions."  Supervisors' job elements require them to
                                 establish, implement, and maintain a quality assurance
                                 program and ensure "...compliance with all quality
                                 assurance related Agency policies, procedures, and
                                 regulations."
PROJECT MANAGERS
DID NOT PROPERLY
APPLY ADDITIONAL
CONTROLS
Although Region 8 strengthened its controls in response to
our 1995 report, the controls did not fully achieve their
intended purposes to prevent untimely submissions and
undocumented plan approvals. Although the Region
implemented a modified sample brokering system to help
eliminate untimely plan submissions, some project managers
did not use the control effectively. Also, project managers
did not always follow Regional policy instituted in 1995
requiring approval signature pages in their plans. Some
project managers neglected to ensure their plans included
signed approval pages.

Six of seven project managers did not use the Region 8
quality assurance group's modified sample brokering system
effectively. The Region implemented a modified sample
                                                                 E1SKG7-08-5001-8100082
                                          14

-------
            Region 8 Needs to Further Strengthen Its Superfund
           	Field Sampling Quality Assurance Controls
brokering system in August 1995 that required the sample
broker to ensure a project manager had an approved project
plan attached to the project manager's request for analytical
services. Although the sample brokering system's primary
purpose was to identify and coordinate Region 8's internal
analytical service requests, the system also provided a
control to help eliminate untimely project plan submissions.
According to the quality assurance staff, the premise was to
have project managers better plan all phases of their
sampling activities. By working backwards, project
managers could determine when they needed their
contractors, other federal agencies, or states to submit the
project plans for EPA's review. Project managers knew
that Regional policy required them to request analytical
services approximately 1-6 weeks prior to sampling and that
approved project plans had to be attached to their request
for analytical services. With this knowledge, project
managers could better project timelines.

However, project managers circumvented this control by
not appropriately using the sample brokering system either
to obtain needed analytical services or provide notification
of external analysis. The project managers allowed their
contractors, other federal agencies, and states to request
sampling analysis outside the Regional sample brokering
system. Project managers  should have provided the sample
broker copies of approved project plans and notified the
sample broker of internal and external laboratory analysis.

Five of seven project managers explained that they use a
tiered approach when managing their contractors and based
their concerns on their level of trust with their contractors.
If a project manager had used a contractor for several years
and was comfortable with the contractor's expertise, the
project manager would not require the contractor to comply
with the 30-day approval rule.  However, as emphasized in
our 1995 report, project managers increased the risk of
inappropriate sampling because contractors may realize that
a final draft plan submitted just prior to a sampling event
                                 E1SKG7-08-5001-8100082
          15

-------
            Region 8 Needs to Further Strengthen Its Superfund
                   Field Sampling Quality Assurance Controls
may not receive a thorough review due to the late
submission.

Four of seven project managers stated they did not
unconditionally require their contractors and potentially
responsible parties to submit plans 30 days in advance.
Project managers said they were not concerned about
having 30 days to review plans because they were
comfortable with the contractors' and potentially
responsible parties' levels of experience and expertise. They
explained they held discussions throughout the project and
believed all parties were in agreement and the project plans
would contain the needed information. Also, project
managers stated they were not concerned that site activities
had significantly changed to warrant a 30-day review
period.

However, the Region wasted money and time on at least
one inappropriate sampling plan that may have been
prevented had the Region required the contractor to follow
the 30-day requirement. The project manager canceled a
sampling event and rejected a contractor-prepared project
plan for the Summitville, Colorado Superfund site because
the plan was inadequate.  However, the project manager
had not received the project plan until the day before the
sampling event and did not review the plan until that time.
Although the contractor never implemented the plan, the
Region was still obligated to pay for it as a deliverable
product.  While the project manager prevented the
contractor from implementing the inadequate sampling plan,
Region 8 paid about $28,000 for this plan that it  could not
use. In addition, Regional staff prepared a "last-minute"
plan for the sampling event. The $28,000 did not include
Regional staff costs.

Some project managers had not diligently implemented the
Region's approval policy and continued to produce plans
that did not demonstrate adequate review and did not have
approval signature pages included within the plan.  We

                                 E1SKG7-08-5001-8100082
         16

-------
            Region 8 Needs to Further Strengthen Its Superfund
           	Field Sampling Quality Assurance Controls
reviewed 8 of the 36 project plans in our sample that were
developed and implemented after our 1995 report.
Although we reported the lack of approval signatures in
1995, we found that 5 of the 8 project plans (63 percent)
still were not signed by the project managers and only 1 of
the 8 project plans received a  quality assurance review. Of
the 5 unsigned plans, 4 (80 percent) did not include
approval signature pages within the plan, while 1 (20
percent) had a signature page  that was not completed. The
Region researched its files and found 2 of the 4 missing
approvals after we issued our  official draft report.
However, neither of the 2 approvals were included in the
project plans as required by Regional policy. In  addition,
the Region approved 1 of the 4 project plans' missing
approvals based on our draft report findings.

Project managers had not ensured that all plans met the
requirement for approval signature pages. Although only
three of seven project managers  stated that they had seen
the Region's 1994 policy memorandum requiring signature
pages, all seven explained that they knew of the approval
requirement. However, we found that they did not
consistently review their contractor-prepared project plans
to ensure a signature page was included. Without signature
pages, we could not easily confirm that plans had been
reviewed or approved by the project manager.  Also, as we
reported in 1995, because there is often  more than one
version of a particular plan, project managers or others
would find it difficult to determine which version was
approved for contractor use for sampling without approval
documentation having been included in  the plan.
                                 E1SKG7-08-5001-8100082
         17

-------
                                              Region 8 Needs to Further Strengthen Its Superfund
                                                     Field Sampling Quality Assurance Controls
ACCOUNTABILITY HAD       Project managers disregarded supervisory suggestions to
NOT BEEN ENFORCED         incorporate the 30-day review period into project planning
                                  schedules and as contract deliverable language for new
                                  contractor work. Project managers continued to allow
                                  contractors and potentially responsible parties to submit
                                  untimely project plans to project managers for review,
                                  comment, and approval without consequence.  Project
                                  managers did not include language in contracts specifying
                                  the 30-day advance plan submission. Although project
                                  managers had penalty provisions available in consent
                                  decrees and other contractual documents, some project
                                  managers had not used these authorities to ensure timely
                                  submissions.  As a result, project managers did always not
                                  hold contractors  or potentially responsible parties
                                  accountable.

                                  Four of seven project managers ignored the 30-day approval
                                  requirement because they did not believe the requirement
                                  was always necessary. Project  managers  explained that
                                  because they had seen previous iterations of the project
                                  plans, the final draft was usually not a surprise. They also
                                  stated that sometimes the Region requested a contractor to
                                  prepare a project plan in less than 30 days to meet urgent
                                  requirements  at a site. They stated that neither project
                                  managers nor contractors should be held accountable for
                                  meeting the 30-day requirement in such cases. However,
                                  supervisors had not confirmed that project managers met
                                  project plan review and approval requirements.

                                  Although Superfund remedial office supervisors' and staffs
                                  performance agreements required quality assurance
                                  compliance, supervisors did not ensure staff complied with
                                  all Regional requirements and staff did not meet all quality
                                  assurance requirements.  Remedial office supervisors stated
                                  they empowered  their staff to perform their duties as "they
                                  see fit" and explained that project managers were
                                  professional staff tasked with many competing priorities.
                                  Supervisors had notified their staff of the Region's policy
                                  requiring documented approvals within project plans and

                                                                   E1SKG7-08-5001-8100082
                                           18

-------
                                             Region 8 Needs to Further Strengthen Its Super-fund
                                            	Field Sampling Quality Assurance Controls
                                  said they believed staff were complying. One supervisor
                                  said that unless he heard of a major problem, he did not
                                  believe there was a problem. As a result, unless notified of
                                  problems, supervisors did not consciously evaluate project
                                  managers' approval actions to ensure they were complying
                                  with the quality assurance requirements.
CONCLUSION
Region 8's strengthened controls did not fully achieve their
intended purposes to prevent untimely project plan
submissions and undocumented plan approvals. The Region
issued a revised policy but did not ensure that the policy
was implemented.  As a result, little actually changed since
our 1995 review. Without the required approval signature
page, the Region could not be certain that the correct plan
version was used and that sampling was done in accordance
with the most current, approved plan.  As a result, the
Region could not ensure it collected all the necessary data
to accurately identify pollution sources and assess risks.
Also, the Region could not ensure it made the most
efficient, effective, or scientifically defensible sampling and
cleanup decisions.  The Region needs to ensure that project
managers comply with the revised Regional policy for
timely submissions and written approval. Top'management
needs to hold staff accountable for properly approving
plans.  Although we agree that staff should be empowered
to perform their duties without undue supervisor scrutiny,
supervisors must ensure that staff meet EPA and Regional
policy requirements.  Top management also needs to hold
contractors and potentially responsible parties accountable
for submitting timely plans.
RECOMMENDATIONS
We recommend the Regional Administrator:

3-1.   Hold Superfund supervisors accountable for
       ensuring that project managers timely and properly
                                                                  E1SKG7-08-5001-8100082
                                           19

-------
            Region 8 Needs to Further Strengthen Its Super-fund
                   Field Sampling Quality Assurance Controls
       review and approve project plans prior to sampling.

3-2.    Implement existing penalty policies regarding late
       deliverables when a pattern of delay becomes
       evident.

3-3.    Hold Superfund supervisors accountable for
       accurately evaluating project managers'  quality
       assurance compliance in their performance
       appraisals.
                                  E1SKG7-08-S001-8100082
         20

-------
                                            Region 8 Needs to Further Strengthen Its Superfund
                                            	Field Sampling Quality Assurance Controls
                                  CHAPTER 4

                        TRAINING WOULD STRENGTHEN
                          QUALITY ASSURANCE SKILLS
                                 Region 8 agreed that project managers needed to attend
                                 mandatory quality assurance training to ensure they had the
                                 necessary skills to properly approve project plans.
                                 However, 13 (or 38 percent) of the project managers and
                                 other staff needing the training had not completed the two
                                 mandatory Regional quality assurance training courses.
                                 Regional policy prescribes two mandatory quality assurance
                                 courses for project managers to ensure they gather sound
                                 data. The Superfund program managers had not regularly
                                 emphasized the importance of quality assurance training or
                                 ensured project managers attended.  Some project managers
                                 believed the training was ineffective and therefore a low
                                 priority. Management should have held project managers
                                 accountable for attending training, and the Regional quality
                                 assurance group needed to improve training effectiveness.
                                 Without well-trained project managers, Region 8 could not
                                 ensure project managers had the necessary skills to properly
                                 review and approve project plans.
EPA AND REGION 8
REQUIRE QUALITY
ASSURANCE TRAINING
EPA and Region 8 policies require certain staff to take
mandatory quality assurance training. EPA Order 5360.1
and Region 8's 1996 Quality Management Plan require all
individuals (staff and their supervisors) directly involved in
data collection activities to have a working understanding of
quality assurance and quality control activities. Specifically,
Region 8's Quality Management Plan requires training for
all project managers with environmental data operation
responsibilities including planning, design, collection, or
assessment activities.  The Quality Management Plan further
suggests that project managers' immediate supervisors I
attend the training.  Region 8's training is provided by the
quality assurance group.
                                                                 E1SKG7-08-5001-8100082
                                          21

-------
                                             Region 8 Needs to Further Strengthen Its Superfund
                                                    Field Sampling Quality Assurance Controls
                                  Regional policy prescribes two mandatory quality assurance
                                  courses for project managers to ensure they gather sound
                                  data. The mandatory courses include a 2-day, 12-hour EPA
                                  course, entitled Regional Quality Assurance Planning
                                  Requirements; and a 1-day, 6-hour EPA course, entitled
                                  Data Quality Objectives. In addition, individuals are
                                  required to take a refresher course once every 2 years. The
                                  Region 8 Quality Management Plan states that these
                                  training requirements were effective on January 19, 1996.
                                  Affected staff had until September 30, 1996, to complete
                                  the required quality assurance training or lose their quality
                                  assurance plan approval authority.

                                  Project managers must take mandatory training for three
                                  reasons: (1) to comply with EPA and Regional policy, (2) to
                                  maintain their project plan approval authority, and (3) to
                                  enhance their skills to serve as project managers.
STAFF HAD NOT
COMPLETED
REQUIRED TRAINING
Although Region 8 Superfund supervisors notified and
encouraged staff to take mandatory quality assurance
training, 13 Region 8 Superfund remedial staff had not
completed the required training. Thirty-five remedial staff
(nine in the Montana office) were subject to the mandatory
quality assurance training including Superfund remedial
program managers, supervisors, project managers, and
Senior Environmental Employees. However,  13 of the 35
staff (38 percent) had not completed the mandatory quality
assurance training including the 2 Superfund remedial
program managers, 9 project managers, and 2  Senior
Environmental Employees.  Seven of the  13 (54 percent),
including the 2 program managers, had not taken any
quality assurance training.

Although the Regional Quality Management Plan does not
require that the two Superfund remedial program managers
take quality assurance training, Region 8 agreed that
"...some familiarity with QA [quality assurance]
                                                                  E1SKG7-08-5001-8100082
                                           22

-------
                                             Region 8 Needs to Further Strengthen Its Superfund
                                            	Field Sampling Quality Assurance Controls
                                 requirements is important for managers...."  Having the two
                                 Superfund program managers attend quality assurance
                                 training would enhance their ability to oversee the
                                 Superfund program and send an important message to staff
                                 that quality assurance is essential and valuable in meeting
                                 program goals.

                                 Nine project managers who had not completed training by
                                 September 30, 1996, were out of compliance with Regional
                                 policy and should have had their project plan approval
                                 authority revoked after that time. Without proper training,
                                 these nine project managers could not ensure they possessed
                                 the skills necessary to appropriately approve project plans.
                                 However, at least two of the project managers continued to
                                 approve plans. Region 8 should have revoked the signature
                                 authority for the nine project managers until the project
                                 managers had completed training. At a minimum,
                                 supervisors should have monitored these project managers'
                                 activities. One supervisor was aware that one project
                                 manager had not completed training and assigned the
                                 "untrained" project manager to work with a project
                                 manager that had completed all training requirements. The
                                 "untrained" project manager completed training in  February
                                 1997 and had not approved project plans between
                                 September 30, 1996, and February 1997. However, the
                                 remaining project managers had not been individually
                                 monitored. Without proper training, project managers
                                 could not ensure they possessed the skills necessary to
                                 approve project plans and, therefore, had not fulfilled their
                                 responsibilities as project  managers.
MANAGEMENT DID
NOT REGULARLY
EMPHASIZE QUALITY
ASSURANCE TRAINING
According to quality assurance staff, Region 8's two
Superfund program managers had not regularly emphasized
the importance of quality assurance training to staff by'
identifying those staff needing training and had not ensured
all staff attended the training. To fulfill Regional quality
assurance training requirements, the quality assurance
                                                                 E1SKG7-08-5001-8100082
                                          23

-------
                                             Region 8 Needs to Further Strengthen Its Superfund
                                                    Field Sampling Quality Assurance Controls
                                  program manager must first identify staff requiring training.
                                  In a December 9, 1996, memorandum to all Region 8
                                  "Program Directors," the quality assurance program
                                  manager requested a list of staff requiring quality assurance
                                  training to be submitted by January 15, 1997.

                                  Although one of the two Superfund program managers had
                                  responded in a previous year to a similar request, neither
                                  responded to the December 1996 request that would have
                                  identified 1997 training needs.  Quality assurance staff said
                                  their perception, based on the Superfund program
                                  managers' unresponsive actions, was that Superfund
                                  program managers did not consider quality assurance
                                  important. Quality assurance staff were concerned that
                                  project managers had reached the same conclusion.
PROJECT MANAGERS
DID NOT VALUE
QUALITY ASSURANCE
TRAINING
Project managers wanted better quality assurance training,
but did not value the current Regional training offered
because they believed the training was ineffective and,
therefore, a low priority. Most of the project managers we
interviewed expressed dissatisfaction with the current
training provided by the Region 8 quality assurance group.
In general, training course evaluations suggested:
(1) providing more coverage of where to get more
information, (2) relating quality assurance to the program
and not the program to quality assurance, (3) providing
laboratory tours to enhance quality assurance
understanding, and (4) discussing more "horror stories" to
make a point of quality assurance importance.

Some project managers said the quality assurance training
was too general and needed to be more program specific.
They explained that some examples used in training did not
accurately depict the problems they often encountered at
their sites because their sites were at different cleanup
phases than the examples used in training. Also, project
managers criticized that presentations were often read from
                                                                  E1SKG7-08-5001-8100082
                                           24

-------
                                              Region 8 Needs to Further Strengthen Its Superfund
                                             	Field Sampling Quality Assurance Controls
                                  overheads and not presented in a manner that would have
                                  enhanced an inexperienced project manager's knowledge.

                                  Some of the Regional quality assurance training, including
                                  the data quality objectives course, did not effectively convey
                                  the underlying quality assurance concepts that would help
                                  project managers better oversee their sites. Project
                                  managers stated that Region 8 quality assurance group
                                  trainers had not linked quality assurance to their program
                                  areas in a manner that enhanced their quality assurance
                                  understanding.  The Regional quality assurance program
                                  manager acknowledged that the group was trying to
                                  improve  its training presentations. He stated that he had
                                  met with staff to discuss alternative presentation approaches
                                  that would provide clearer and more relevant information to
                                  program staff. He had also met with other Regional
                                  program managers to determine specific staff needs and
                                  planned to meet with other managers to assess training
                                  needs.
CONCLUSION                   Thirteen (38 percent) of Region 8's Superfund remedial
                                  program staff had not completed mandatory quality
                                  assurance training, 9 of which were project managers with
                                  the responsibility to review and approve project plans.
                                  Without well trained project managers, Region 8 could not
                                  ensure project managers had the necessary skills to properly
                                  review and approve project plans. We believe a direct
                                  correlation exists between senior management emphasizing
                                  the importance of quality assurance and project managers
                                  attending training. When the two Superfund program
                                  managers did not respond to the quality assurance training
                                  request, the message delivered to the quality assurance
                                  group and, possibly to project managers, was that quality
                                  assurance is unimportant.  The Region needed to ensure
                                  attendance policies are adhered to by all affected staff and
                                  that training is effective.
                                                                   E1SKG7-08-5001-8100082
                                           25

-------
                                              Region 8 Needs to Further Strengthen Its Superfund
                                                     Field Sampling Quality Assurance Controls
RECOMMENDATIONS          We recommend the Regional Administrator:

                                  4-1.   Direct the Superfund remedial program managers to
                                         identify staff who have not completed required
                                         training and hold staff accountable for attending.

                                  4-2.   Involve program and project managers in the design
                                         and timing of quality assurance training.

                                  4-3.   Direct the quality assurance staff to routinely solicit
                                         customer comments from program staff and evaluate
                                         comments to improve services.

                                  4-4.   Direct Superfund program staff to promote quality
                                         assurance through initiatives such as inviting quality
                                         assurance staff to attend Superfund program staff
                                         meetings to explain the benefits of quality assurance
                                         to the Superfund program.

                                  4-5.   Obtain training for the quality assurance staff in
                                         presentation techniques to help quality assurance
                                         staff better articulate how quality assurance adds
                                         value.
                                                                    E1SKG7-08-5001-8100082
                                            26

-------
                                             Region 8 Needs to Further Strengthen Its Superfund
                                            	Field Sampling Quality Assurance Controls
                                  CHAPTER 5

               FURTHER ACTIONS COULD ENSURE APPROPRIATE
                      AND WELL-DOCUMENTED OVERSIGHT
                                 Region 8 had inadequate assurance that data collected
                                 during the remedial cleanup phase was reliable and
                                 defensible because project managers had not significantly
                                 changed their oversight practices to ensure compliance with
                                 project plans. Regional policy, project managers' position
                                 descriptions, and performance agreements outline oversight
                                 responsibilities and good management practices. However,
                                 some project managers continued to perform limited
                                 oversight, relied too heavily on oversight contractors, and
                                 did not use the Region 8 quality assurance group's
                                 assistance. Also, project managers had not appropriately
                                 documented their oversight activities. Project managers had
                                 no procedures specifically describing their oversight and site
                                 management responsibilities and authority. Project
                                 managers needed more specific direction and clearer
                                 authority.  When project managers rely too heavily on
                                 contractors and do not observe contractor activities, they
                                 cannot ensure contractors perform as intended, are fully
                                 knowledgeable of site activities, or enhance their own
                                 experience. Without appropriate and documented
                                 oversight, Region 8 could not ensure data was reliable and
                                 defensible.
PROJECT MANAGERS'
RESPONSIBILITIES
INCLUDE OVERSIGHT
Regional policy outlines project manager oversight
responsibilities and good management practices. As a result
of our prior review, the Region issued a 1995 policy
memorandum that provided some general suggestions to
project managers on contractor oversight and identified
other technical personnel available to assist them. The
policy memorandum emphasized that oversight of
contractors was  one of project managers' most important
responsibilities and that project managers should use
available Regional expertise to review workplans. The
                                                                 E1SKG7-08-5001-8100082
                                          27

-------
                                              Region 8 Needs to Further Strengthen Its Superfund
                                                     Field Sampling Quality Assurance Controls
                                  memorandum also instructed project managers to evaluate
                                  contractor performance, such as through field audits.  The
                                  policy memorandum also directed project managers to
                                  document activities in "records of communication" as a
                                  good management practice and to periodically observe field
                                  sampling activities.

                                  Project manager oversight responsibilities had not changed
                                  since our 1995 review and are addressed in generic position
                                  descriptions and performance agreements. Project manager
                                  position descriptions at General Schedule levels 9-13 still
                                  included several statements applicable to oversight
                                  activities, including:  (1) ensuring technical soundness of
                                  remedial efforts accomplished by contractors, states, and
                                  potentially responsible parties; and (2) directing, overseeing,
                                  managing, and evaluating contractor actions.  Generic
                                  performance agreements specifically required project
                                  managers to oversee compliance with orders and decrees
                                  and take appropriate action in cases of noncompliance;
                                  monitor contractor performance through invoices and
                                  supporting documentation; maintain appropriate records;
                                  communicate clearly, concisely, timely, and responsively
                                  with colleagues and managers; and prepare written work
                                  and presentations  that are clear, relevant, concise, well-
                                  organized, and appropriate to the  audience.
OVERSIGHT
ACTIVITIES HAD NOT
SIGNIFICANTLY
IMPROVED
Although we criticized Region 8 project oversight in our
1995 review, project managers did not significantly revise
their oversight activities to ensure contractors complied
with project plans and site history files supported decisions
and justified resources. Project managers continued to rely
too heavily on oversight contractors and six of seven had
not appropriately documented their oversight activities.
Project managers did not periodically observe contractor
field activities and, therefore, could not independently
confirm contractors complied with project plans. Also,
                                                                   E1SKG7-08-5001-8100082
                                           28

-------
            Region 8 Needs to Further Strengthen Its Superfund
           	Field Sampling Quality Assurance Controls
some project managers did not maintain adequate site
history files that would support site management decisions.

Some project managers did not revise their oversight
activities. Three of seven project managers continued to
rely on their field sampling and oversight contractors to
conduct sampling and provide oversight of other
contractors and did not personally observe field sampling
activities as directed in the Regional policy memorandum.
Although project managers received contractor- or state-
prepared progress reports to help them monitor site
activities, they did not regularly schedule site visits to
observe contractor performance or sampling activities. As
such, project managers could not ensure contractors
performed as expected in workplans, and project managers
did not broaden their experience and knowledge about their
sites' sampling activities. As we reported in 1995,
inadequate oversight increased the risk that contractors
might not be performing in accordance with statements of
work and workplans. Also, contractors could have
circumvented sampling requirements since project managers
did not periodically observe contractors' field activities.

Six of seven project managers had not appropriately
documented their oversight actions and maintained a
complete site history file to support field sampling and other
decisions, resources expended, and site transitions between
project managers or between EPA and states. One project
manager voiced concern about assuming responsibility for a
site that had an incomplete site history file because a
previous project manager did not prepare memorandums-of-
record to document what had transpired at the site. The
project manager explained that file documentation should
provide evidence that supports: (1) what actions occurred,
(2) what agreements were reached, (3) how funds were
spent, and (4) how resources were used when transitioning
a site from one project manager to another or when
transitioning a site from EPA to the state.  Without
documentation that supported oversight actions, newly
assigned project managers were less efficient and effective.

                                 E1SKG7-08-5001-8100082
         29

-------
                                             Region 8 Needs to Further Strengthen Its Superfund
                                                    Field Sampling Quality Assurance Controls
OVERSIGHT
RESPONSIBILITIES AND
AUTHORITY WERE
UNCLEAR
Project managers had no procedures clearly describing their
oversight and site management responsibilities and
authority. The Regional policy memorandum was too
general, and did not provide specific oversight procedures
that directed project managers' actions and site management
activities. Because some project managers did not believe
they had specific procedures, they stated they performed
their project manager responsibilities in accordance with
other directives, such as consent decrees. They explained
that consent decrees provide specific requirements or
actions, such as defining documentation requirements and
stipulating timelines for construction activities.  Project
managers explained that they wanted to maintain their
authority to apply their professional judgment when making
site decisions but would welcome better direction. They
explained that part of their reliance on contractors resulted
from competing priorities and insufficient time to complete
all their work. They also explained that they knew they had
authorities to enforce contractor compliance with workplans
but were unsure whether they would get management
support if they tried to take action against a contractor for
poor performance.

Some project managers explained that they used a tiered
approach when managing contractors. They stated that if
they had extensively used a contractor and had established
confidence in the contractor's  abilities, the project managers
would allow the contractor more discretion and not exercise
as much oversight as would be necessary if the contractor
was inexperienced or new to the site.

Program supervisors did not ensure staff properly executed
oversight responsibilities.  One supervisor stated that he
allowed his staff to use their professional judgment as to site
visits and what information needed to be documented. He
believed his staff was fulfilling his expectations.  Another
supervisor explained that he counseled staff about
documentation requirements but did not believe in micro-
managing staff. Both supervisors believed their staff was
appropriately performing oversight.
                                                                  E1SKG7-08-5001-8100082
                                           30

-------
                                             Region 8 Needs to Further Strengthen Its Superfund
                                                    Field Sampling Quality Assurance Controls
PROJECT MANAGERS
DID NOT VALUE
QUALITY ASSURANCE
ASSISTANCE
Project managers did not value the quality assurance
group's assistance and believed that the quality assurance
group was unresponsive to their needs because the group's
comments on project plans were not always constructive.
Also, they believed the quality assurance group's
implementation of quality assurance rules was often too
rigid. They stated the group needed to be more customer-
oriented. As a result, project managers did not routinely
solicit the group's input on project plans or consistently use
the Region's sample brokering system. The Regional
quality assurance program manager acknowledged that the
quality assurance group had a negative image.

Some project managers did not consistently use the quality
assurance group to assist them during their reviews because
they did not consider quality assurance comments to be
constructive or value-added.  Project managers stated that
some quality assurance staff did not understand project
managers' needs. Project managers also said they believed
quality assurance staff required them to include information
in plans that was not relevant to the particular site.

However, some project managers provided very positive
feedback about one quality assurance staff member who had
a strong project manager background. On the strength of
her background, this quality assurance staff member
understood project managers' needs and provided timely,
effective feedback. For example, she explained "line by
line" any revisions to project plans that she believed were
necessary.  According to one  project manager, when this
staff member left the quality assurance group, subsequent
quality assurance comments on project managers' plans did
not provide the same specificity. Subsequent quality
assurance comments were sometimes generic and required
certain formats be followed before the quality assurance
group would agree with the project plan contents. Asa
result, project managers did not readily accept subsequent
quality assurance comments and believed quality assurance
requirements were too rigid.
                                                                  E1SKG7-08-5001-8100082
                                          31

-------
                                              Region 8 Needs to Further Strengthen Its Superfund
                                                     Field Sampling Quality Assurance Controls
                                  Project managers explained, and the quality assurance group
                                  agreed, that the sample brokering system was sometimes
                                  rigid and not user-friendly. For example, one project
                                  manager experienced problems scheduling analytical
                                  services. Regional policy required written requests for
                                  analytical services. However, the sample broker verbally
                                  authorized the project manager's request, but the Region 8
                                  laboratory had not received the authorization and would not
                                  analyze her samples without a written authorization. The
                                  project manager had to obtain funding to use a commercial
                                  laboratory because the Region 8 laboratory would not
                                  analyze the samples on such short notice. The perception
                                  left with the project manager was that the requirement for a
                                  written authorization was too rigid. The project manager
                                  stated that other project managers also resist having to use a
                                  rigid sample brokering system or being told they must use
                                  this system. The quality assurance group acknowledged
                                  some scheduling problems and was aware that it needed to
                                  be more flexible. The sample broker and the quality
                                  assurance program manager have met with project
                                  managers to discuss how to improve the process.

                                  The Regional quality assurance program manager
                                  recognized that the quality assurance group's image was
                                  negative. He  explained that the group had been working
                                  closer with program offices to change project managers'
                                  perceptions and help meet their program goals.  Because the
                                  quality assurance group can be instrumental in helping
                                  project managers effectively perform their jobs, the quality
                                  assurance group must ensure its staff effectively
                                  communicate to promote quality assurance practices.
CONCLUSION                   Regional project managers did not properly implement
                                  Region 8's controls or improve their oversight activities.
                                  Project managers had not significantly changed their
                                  oversight practices to ensure compliance with project plans.
                                  When project managers rely too heavily on contractors and


                                                                   E1SKG7-08-5001-8100082
                                           32

-------
                                              Region 8 Needs to Further Strengthen Its Superfund
                                             	Field Sampling Quality Assurance Controls
                                  do not observe contractor activities, they cannot ensure
                                  contractors performed as intended, be fully knowledgeable
                                  of site activities, or enhance their own experience. Without
                                  appropriate and documented oversight, Region 8 cannot
                                  ensure data is reliable and defensible. Controls may need to
                                  be reexamined to provide for improved contractor oversight
                                  and more responsible project manager actions.
RECOMMENDATIONS          We recommend the Regional Administrator:

                                  5-1.    Hold project managers accountable for following
                                         existing Region 8 policy and determine whether
                                         additional oversight direction is needed.

                                  5-2.    Schedule and budget for a minimum number of site
                                         visits to observe contractors' field sampling
                                         activities and oversight of other contractors.

                                  5-3.    Specify the type of site file documentation (e.g.,
                                         agreements reached) required in site files to fully
                                         document field sampling decisions.

                                  5-4.    Assure staff that management will support staff
                                         decisions to take appropriate action if contractors
                                         perform poorly.

                                  5-5.    Determine whether project managers should be
                                         required to obtain Regional quality assurance
                                         assistance before approving  a project plan.

                                  5-6.    Provide sample brokering standard operating
                                         procedures to project managers.
                                                                   E1SKG7-08-5001-8100082
                                           33

-------
                                          Region 8 Needs to Further Strengthen Its Superfund
                                                Field Sampling Oualitv Assurance Controls
                                                                     EXHIBIT 1
               THE DATA QUALITY OBJECTIVES PROCESS
                COMPARED TO THE SCIENTIFIC METHOD
The scientific method and the data quality objectives process provide tools to plan activities.
EPA's quality system is based on the scientific method which consists of three stages: planning,
implementation, and assessment. The 7-step data quality objectives process is comparable to the
planning stage of the scientific method (steps 1-4).  We compared the planning stage of the
scientific method to the data quality objectives process. The following tables correlate the steps
of the scientific method with those of the data quality objectives process.
          SCIENTIFIC METHOD

  1.    DEFINE A PROBLEM.
  2.    GATHER PERTINENT
         HISTORICAL/BACKGROUND
         INFORMATION.
  3.    FORM A WORKING HYPOTHESIS OR
         EXPLANATION.
  4.    DO EXPERIMENTS TO TEST THE
         HYPOTHESIS.
  5.    INTERPRET THE RESULTS.
  6.    DRAW A CONCLUSION AND
        MODIFY THE HYPOTHESIS.
 DATA QUALITY OBJECTIVES PROCESS

1.     STATE THE PROBLEM.
2.     IDENTIFY THE DECISION.
3.     IDENTIFY INPUTS TO THE DECISION.
4.     DEFINE THE STUDY BOUNDARIES.
5.     DEVELOP A DECISION RULE.
6.     SPECIFY LIMITS ON DECISION
       ERRORS.
7.     OPTIMIZE THE DESIGN FOR
      OBTAINING DATA.

Scientific
Method
Data Quality
Objectives
STEPS
1

1,2

2

3

3

5,6

4

*4,7

5

—

6

—

* Data quality objectives steps 4 and 7 should be continually assessed throughout a project.
                                                             E1SKG7-08-5001-8100082
                                       34

-------
                                             Region 8 Needs to Further Strengthen Its Superfund
                                             	Field Sampling Quality Assurance Controls
                                                                           EXHIBIT 2

                         SCOPE AND METHODOLOGY
We conducted our fieldwork from January 1997 through November 1997 in accordance with
Government Auditing Standards (1994 Revision) issued by the Comptroller General of the United
States. We designed our followup audit to gather information that would determine whether the
Region's corrective actions improved its field sampling quality assurance activities after issuance
of our January 1995 report.

We judgmentally selected 10 (approximately 25 percent) Region 8 Superfund sites from EPA
Region 8's National Priority JJst. Region 8's list included 5 states with a total of 38 general
Superfund sites and 9 federal facilities sites.  We did not include the federal facilities sites within
our universe. We judgmentally selected two sites from each of the five states except South
Dakota and Colorado. South Dakota only listed two sites of which one was a federal facility site.
In place of the South Dakota site, we selected a third site from Colorado, as Colorado had the
majority of sites listed.  We judgmentally selected the sites based on size, money spent, and time
listed on the National Priority List. We judgmentally selected 36 project plans for these sites that
included sampling and analysis plans, quality assurance project plans, and field sampling plans.
We selected the plans based on site index sheets provided by the Superfund Records Center. We
evaluated the project plans to determine  whether Region 8 adequately implemented and
documented its data quality objectives process.

We reviewed Regional policy memorandums issued in response to our 1995 report and examined
the same 36 project plans to determine whether project managers had complied with the Region's
additional controls. Because we were interested hi Regional improvements after our 1995 report,
we narrowed those plans to 8 that the Region developed and implemented after our 1995 report.
The additional Regional controls included having signed signature pages within the project plans
and requiring all analytical service requests to go through the Region's sample brokering system.
Also, we interviewed 7 project managers that managed 8 of the 10 sites hi our review and 2
Superfund remedial program supervisors about their awareness of the additional controls and
staffs' implementation of the Region's revised policies.

We interviewed the same seven project managers and supervisors about their mandatory quality
assurance training attendance and understanding of quality assurance concepts, especially related
to the data quality objectives process. We relied on training records from the regional quality
assurance program manager, the remedial program training coordinator, and the Montana training
coordinator to determine whether the appropriate project managers and other remedial staff had
attended training. Region 8 changed its training requirements from its 1992 to 1996 Quality

                                                                  E1SKG7-08-5001-8100082
                                           35

-------
Management Plans. We used the Region's 1996 Quality Management Plan to assess staffs
compliance with training requirements.

We reviewed Regional policy memorandums issued in response to our 1995 report. We
interviewed the same seven project managers about their oversight responsibilities, available
guidance, and implementation of revised Regional policy memorandums to determine whether
project managers had changed their past oversight practices in accordance with the revised
Region 8 policy.
                                                                  E1SKG7-08-5001-8100082
                                           36

-------
                                             Region 8 Needs to Further Strengthen Its Superfund
                                            	Field Sampling Quality Assurance Controls
                                                                          EXHIBIT 3

                  EPA COMMENTS AND OIG EVALUATION
Region 8 generally concurred with our recommendations and presented an action plan that
outlined how the Region planned to address our recommendations. The action plan generally
satisfied the intent of our recommendations. We agree that Region 8 had improved its field
sampling quality assurance controls since our last report. We also agree that the Region can
further improve its quality assurance efforts in its Superfund field sampling planning.  We
reviewed Region 8's specific comments to our draft report and made some minor modifications to
the final report as we determined necessary.

Region 8 made some comments for which we made no changes to the report. We did not revise
our final report to include Region 8 comments about the basin-wide quality assurance project plan
because we still conclude that the site-specific sampling and analysis plan needed specific data
quality objectives. We also did not change our position on the sample brokering system. We
agree that the Region cannot direct contractors to select a specific subcontractor for laboratory
analysis.  However, the Region may direct contractors to use the Region 8 laboratory for analysis
which would require use of the sample brokering system. Furthermore, Regional policy requires
project managers to notify the sample broker before any sampling is done and requires project
managers to provide the sample broker with an approved sampling plan regardless of who is
doing the laboratory analysis. We also did not revise our discussion of the untimely Summitville
sampling and analysis plan because we believe it exemplifies the issue of timely plan review. We
also did not change our position on inadequate oversight because inappropriate levels of oversight
have persisted since our last audit.

See Appendix I for the full Agency response.
                                                                 E1SKG7-08-5001-8100082
                                          37

-------
          UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
                             REGION VIII
                       999 18th STREET - SUITE 500
                      DENVER, COLORADO 80202-2466
                                              APPENDIX  I
                                              Page  1  of 5
                            FEB 1 3 1998
 Ref:   8TMS-G
MEMORANDUM
SUBJECT:
FROM:
TO:
Comments on Draft Report, Region 8 Needs  to  Further
Strengthen Its Superfund Field Sampling Quality
  ssurance Controls,Audit Report E1SKG7-08-5001-XXXXXXX
            elldtotai
Regional Administrator
1
Bennie S. Salem
Divisional Inspector General For Audit
Central Audit Division
     As  requested in the December 23,  1997,  cover memorandum to
the above-referenced report,  we are providing comments on the
findings of  the report,  a concurrence  or conditional concurrence
with each  of the recommendations presented,  and an action plan
that addresses  each of  the recommendations.

     We  agree that further improvements can be made to quality
assurance  efforts in our Superfund field sampling planning.
However, we  also believe that significant improvements have  been
made since the  previous  Office of Inspector General (OIG)  report
issued in  January 1995 and that project managers are implementing
the program  in  a sound and responsible manner.  We believe the
draft report unfavorably describes our quality assurance program,
drawing  overall conclusions based on very narrow instances where
we have  fallen  short.  We suggest that more  recognition be made
of the actions  taken in  response to the previous report.   We do
appreciate the  suggestions made by the OIG on how improvements
could be made in our current  processes and program controls.

     Despite our concerns with the overall project approach  and
the manner in which the  findings are presented,  we are planning
to address the  recommendations made, with the conditions listed
below.   An action plan is attached that provides information on
how we plan  to  address those  recommendations.
                                    38
                                                         Printed on Recycled Paper

-------
                                                        APPENDIX  I
                                                        Page  2  of 5


      Our response to the report recommendations is as follows:

 2-1  Concur,  provided that the last sentence of the recommendation
 is deleted.   See Action Plan.               ,(

 2-2  Concur.   See Action Plan.

 2-3  Concur.   See Action Plan.

 2-4  Concur.   See Action Plan.

 3-1  Concur.   See Action Plan.

 3-2  Concur.   See Action Plan.

 3-3  Concur,  provided that the  recommendation is modified to state
 something such as, "When a pattern of delay becomes evident,
 implement existing policies  regarding  late deliverables at a
 minimum."  See Action Plan.

 3-4  We understand that  this  recommendation is being deleted and
 will not appear  in the  final report.

 3-5  Concur.   See Action Plan.

 4-1  Concur.   See Action Plan.

 4-2  Concur.   See Action Plan.

 4-3  Concur.   See Action Plan.

 4-4  Concur.   See Action Plan.

 4-5  Concur.   See Action Plan.

 5-1  Concur.   See Action Plan.

 5-2  Concur.   See Action Plan.

 5-3  Concur,  provided that  the  phrase,  "and other site decisions,"
 is deleted.

 5-4  Concur.   See Action Plan.

 5-5 Concur.   See Action Plan.

 5-6 Concur.   See Action Plan.

     Please  note that specific comments on the report are  listed
 in Attachment 1.  The Action Plan  is described in  Attachment 2.
We would like to thank  you for the opportunity to  provide
 comments on  the  draft report.   For questions,  please  call  Barbara
Rodriguez, Regional  Audit Coordinator, at  (303)  312-6360.
                                    39
                                                        Printed on Recycled Papar

-------
                                                       APPENDIX I
                                                       Page 3 of 5


                           Attachment 1


                    Specific Comments on Report


 Chapter 2:   Throughout  this  chapter, it  should be made clear  that
 project managers  have had  the  responsibility for approval of
 quality^assurance documents  only since 1992.   Prior  to that time,
 the  Regional Quality Assurance Officer had  the responsibility.
 For  example,  the  first  sentence on p. 7  should be clarified.

 p. 7,  last  paragraph.   The Quality Assurance Project Plan
 developed for the entire Clark Fork Basin was approved on
 May  20,  1992.   It addressed  DQO's in an  adequate manner with  the
 understanding that each subsequent SAP would include any site-
 specific DQO's as appropriate.   Much of  the design work included
 in the Streamside Tailings Operable Unit was to further
 characterize the  area as an  extension of the RI/FS,  and thus  the
 decision was made that  the  basin-wide QAPP should be the
 governing plan.   The State of  Montana is now the lead for
 construction of the remedy.  DQO's for that aspect of the remedy
 should be part of the planning process for  the state's work under
 a cooperative agreement.

 p. 11,  2nd  paragraph.   The sample brokering program  was designed
 only for "analytical work to be performed for all EPA Region VIII
 programs  ...  through lAGs, States, and contractors."
 Consequently it does not apply to PRP-lead  sites. Also, the
 brokering system  does not  apply to those instances where
 laboratory  services were being subcontracted out.  That
 understanding is  reflected in  the Regional  Quality Management
 Plan.   The  brokering system  thus  applies only to situations where
 EPA  itself  or EPA through  an IAG or contract  is  directly
 responsible for obtaining  analytical services  through the
 Regional  Laboratory, the Contract Laboratory  Program, or other
 vehicle.

 p. 11,  2nd  paragraph. The  sample  broker does  not review each
 sampling  plan.  He  only knows  that the plan is approved.   He then
 receives-  a  copy of  the analytes.

 P. 12,  1st  full paragraph.   This  example of the  Summitville
 sampling  plan  does  not demonstrate a situation that would be
 helped by requiring a 30-day review period.  The actual situation
 is that the contractor was tasked with preparing a sampling plan.
Generally these plans are prepared with significant data user
 input. However, in  this case the  EPA data user was not able to
 spend the time necessary to allow preparation of an adequate
document.  Thus when the draft plan came in, there had been a
project management  change that resulted in a new data user
                                    40

-------
                                                       APPENDIX I
                                                       Page 4 of 5
                                                                2

becoming  involved.  This new person decided that the plan did not
meet  EPA  needs and asked that another plan be prepared. It was
estimated that the costs incurred to date were $28,000.  We do
not believe  that a 30-day requirement on a draft plan could have
helped a  situation that was directly related to the data user
needs changing toward the end of the planning effort.

p. 12, 2nd paragraph. First sentence should begin,  "Some project
managers," rather than the blanket statement,  "Project managers"
which makes  the reader think that all or a majority of, project
managers  were not diligent in carrying out their duties.

p. 12, 2nd paragraph.  We have searched our records and have
found approval letters for two of the five plans that did not
have  approval signatures.  Approval documentation for these is
attached  as  Attachments 1-A and 1-B.  A third plan, for Butte
Priority  Soils, was approved based on the findings in the OIG
draft report (Attachment 1-C).  These approval letters were not
attached  to  the sampling plans.  One additional plan, for
Streamside Tailings, was never implemented because EPA took back
the work  and gave it to the State.  Therefore, the State would
have  the  responsibility, for approving the plan.  Thus, there is
only  one  plan that has not been approved.           ->

p. 13, top of page.  Region 8 already has in place mechanisms for
holding PRPs and contractors accountable.  PRP work is done under
orders or decrees that specify the work to be done.  In
agreements there are stipulated penalties for late submission of
deliverables.  In unilateral orders, there is the threat of
enforcement.  Penalties are determined by a number of factors and
are at the discretion of the project coordinator and signatory.
For contracts the contract document itself describes the
procedure  to be followed in cases of lack of performance.  Work
assignments  are evaluated and awards can be reduced by the
Performance  Evaluation Board based on the criteria in the
contract.

p. 16, 3rd paragraph.  Attachment 1-D is a table that summarizes
our findings on QA training needs.  We believe that seven project
managers, including a new-hire, currently require at least some
training.  As of September 30, 1997, we believe there were eight
project managers needing training.  One person, Ken Wangerud, who
was identified in the report as needing training, has been
exempted  from some training according to the Rick Edmonds
transition plan memo of January 9, 1996.  Another, Lisa Lloyd,
completed her training after September 30.  Those who are
currently untrained have been counseled about not signing plans
until they are fully trained.  We agree that the two SEE
employees should be trained.  Although not required by the
Regional QAMP, we agree that some familiarity with QA
requirements is important for managers of project managers.  The
                                    41

-------
                                                       APPENDIX I
                                                       Page 5 of 5
QA Program will be developing a short course specific to this
group.

p. 18.  first sentence as continued from p. 17.  We disagree with
the statement that managers are delivering a message that QA is
not important, using the example of not responding to the request
for information.  Without supporting information, it is
presumptuous of the OIG to conclude that the reason for not
responding is a view that QA is not important.  Even though there
may be a perception of this view by some people in the QA Office,
in fact, a memo was distributed to all RPMs in the Superfund
Remedial Response Program from the Program Director on January
16, 1996, that clearly states the importance of receiving this
training.  This was one week after the transition plan memo was
issued.

p. 20, second paragraph under "Oversight..."  It is not clear how
widespread this problem is.  We believe there may be a few
project managers who may not have been present at their sites
during a field sampling episode. However, project managers
routinely perform site visits for one purpose or another and
personally, or with state staff assistance, oversee contractor or
PRP work.  Site-specific travel is routinely approved and
encouraged as needed to assure proper oversight.

p. 20, third paragraph under "Oversight..."  Same as above.
                                    42

-------
                                            Region 8 Needs to Further Strengthen Its Superfund
                                           	Field Sampling Quality Assurance Controls
                                   APPENDIX II

                                 DISTRIBUTION
Office of Inspector General
       Inspector General (2410)
       Deputy Assistant Inspector General for
        Internal Audits (2421)
       Headquarters Audit Liaison (2421)
       Divisional Inspectors General for Audit

Headquarters Office
       Assistant Administrator for Solid Waste and Emergency Response (5101)
       Director, Office of Emergency and Remedial Response (5201)
       Division Director, Analytical Operations and Data Quality Center (5202G)
       Division Director, Regions 5/7 Accelerated Response Center (5202G)
       Assistant Administrator for Research and Development  (8101)
       Director, National Center for Environmental Research and Quality Assurance (8701)
       Division Director, Quality Assurance Division (8724R)
       Agency Followup Official (2710)
       Agency Followup Coordinator (3304)
       Associate Administrator for Regional Operations and State/Local Relations (1501)

EPA Region 8 Office
       Assistant Regional Administrator for
        Ecosystems Protection and Remediation (8EPR)
       Assistant Regional Administrator for
        Technical and Management Services (8TMS)
       Director, Montana Operations Office (8MO)
       Program Manager, Superfund Remedial Response - Denver (8EPR-SR].
       Program Manager, Superfund Remedial Response - Montana (8MO)
       Program Manager, Quality Assurance (8TMS-Q)
       Audit Coordinator (8TMS-G)
                                                                E1SKG7-08-5001-8100082
                                          43

-------