f
I
C3
Office of Inspector General
Audit Report
Superfund
Environmental Data Quality at
Superfund Removal Actions
in Region 9
El SFF7-09-0058-8100223
September 4,1998
Tucson International Airport PCS Removal
-------
Inspector General Division Western Audit Division
Conducting the Audit: Sacramento Branch Office
Region Covered: Region 9
San Francisco, California
Divisions Involved: Superfund Division
Policy and Management Division
Cover Photo: Tucson International Airport PCB Removal
Tucson, Arizona
Photograph by Dan Cox, EPA OIG
-------
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
OFFICE OF THE INSPECTOR GENERAL FOR AUDITS
WESTERN DIVISION
75 HAWTHORNE STREET
19TH FLOOR, MAIL CODE M
SAN FRANCISCO, CA 94105-3901
September 4, 1998
MEMORANDUM
SUBJECT: Report on Environmental Data Quality at Superfund Removal Actions
in Region 9
Audit Report No. E1SFF7-09-0058-8100223
FROM: Truman R.
Divisional Inspector General for Audit
Western Division
TO: Felicia Marcus
Regional Administrator
Region 9
Attached is our final report on Environmental Data Quality at Superfund Removal Actions in
Region 9. The purpose of the audit was to determine if Region 9 had sufficient procedures in
place to ensure that environmental data was of known and acceptable quality for Superfund
removal actions.
Our audit of five removal actions showed Region 9 did not have sufficient procedures over
Superfund removal actions to ensure that environmental data was of known and acceptable
quality. Also, the Region did not fully use EPA's scientific planning process, called data quality
objectives, to ensure its removal actions and corresponding data collection activities were
effective and efficient.
This audit is also part of a national audit of field sampling. Accordingly, the results of this audit
will also be included in a national audit report, to be issued in 1998.
This audit report contains findings that describe problems the Office of Inspector General has
identified and corrective actions the OIG recommends. This audit report represents the opinion
of the OIG and the findings contained in this audit report do not necessarily represent the final
EPA position. Final determinations on matters in this report will be made by EPA managers in
accordance with established EPA audit resolution procedures. Accordingly, the findings
described in this report are not binding upon EPA in any enforcement proceeding brought by
EPA or the Department of Justice.
Primed on Recycled Paper
-------
ACTION REQUIRED
In accordance with EPA Order 2750, you, as the Action Official, are required to provide us with
a written response to this report within 90 days of the report date. For corrective actions planned
but not completed by the response date, we need specific milestone dates to decide whether to
close this report.
We have no objection to the further release of this report to the public.
If you have any questions, please contact Charles McCollum, Audit Manager, at (916) 498-6530,
or Katherine Thompson, Auditor in Charge, at (916) 498-6535.
Attachment
Distribution: Appendix G
-------
EXECUTIVE SUMMARY
INTRODUCTION
OBJECTIVE
RESULTS IN
BRIEF
EPA's Emergency Response Program responds to threats posed by
the sudden or unexpected releases of hazardous substances.
Region 9's Emergency Response Office manages the majority of
these responses, called removal actions, throughout California,
Nevada, Arizona, Hawaii, and the Pacific Islands.
We performed this audit because of known risks to the quality of
environmental data. Region 9's Superfund Program has
experienced serious problems with environmental data quality that
are likely to continue. Risks to environmental data are significant
because data is the basis for EPA's decision making and
enforcement actions.
This audit is also part of a national audit of field sampling. The
results of this audit pertaining to field sampling will be included in
a national audit report.
The objective of the audit was to determine if Region 9 had
sufficient procedures in place to ensure that environmental data
was of known and acceptable quality for Superfund removal
actions.
Our audit of five removal actions
showed Region 9 did not have
sufficient procedures over Superfund
removal actions to ensure that
environmental data was of known and
acceptable quality. Also, the Region
did not fully use EPA's planning
process, called data quality objectives,
to ensure its removal actions and
corresponding data collection activities were effective and
efficient.
Actions Audited
Casmalia
Cruz Ranch
Dodson Brothers
Sanders Aviation
Tucson Airport
-------
Chapter 1: Quality
System for
Removal Actions
Needs
Improvement
Chapter 2: Data
Quality Objectives
Could Improve
Removal Planning
and Data
Collection
Our audit of five removal actions showed the Region did not have
a quality management plan which adequately documented and
described the quality system for removal actions. Further, most of
the site-specific quality assurance project plans (QAPPs) we
reviewed:
• Were not based on the seven-step data quality objectives
(DQO) process;
• Were not designed to prevent and detect inappropriate quality
data;
• Did not include defensible or optimal plans for collecting data;
• Were not reviewed or approved by the Region's Quality
Assurance Office; and,
• Were not implemented or monitored.
We believe the main reason the quality system was insufficient was
because decision makers did not perceive data quality as a risk to
decision making.
Consequently, the Region completed three removal actions without
appropriate quality data for decision making. In addition, about
420 samples were analyzed for one removal action that were not
used for the decision indicated in the sampling plan.
The Region undertook removal actions and corresponding data
collection efforts without fully using the data quality objectives
process, EPA's systematic planning method.
The DQO process provides two primary benefits: (1) better
decisions, because they are based on the scientific method; and (2)
more cost-effective sampling for environmental data.
The Region did not use the
complete DQO process at
the five removal actions
we audited because on-
scene coordinators were
not required to and they
were generally unfamiliar
with the process. Also, the
Region did not have a
system in place to support the process.
"Data cannot be evaluated as
good or not good until the use of
that data has been clearly
defined."
-Principles of Environmental
Sampling, Second Edition, 1996
11
-------
POSITIVE ACTIONS
TAKEN
As a result, the Region completed five removal actions, costing
more than $20 million, without sufficiently documenting important
decision criteria or alternatives.
The five Superfund removal actions we audited are described in
more detail in Appendix B. Appendix C explains the abbreviations
we used. Appendix D lists the criteria applicable to data quality
and removal actions.
Although we found deficiencies in the quality assurance system for
removal actions, we noted that:
• The Region had a quality management plan that was approved
by EPA's headquarters;
• The Region's quality management plan required QAPPs to be
approved by its Quality Assurance Office;
• The Region's Emergency Response Office had established a
"generic" QAPP that provided quality assurance guidance for
its removal actions;
• The Emergency Response Office's technical contractor had a
quality management plan that required significant quality
assurance activities;
• The Emergency Response Office and Superfund Cleanup
Branch had prepared site-specific QAPPs for all the removal
actions we audited; and,
• All of the site-specific QAPPs we reviewed required the data to
be validated, an important quality assurance activity.
RECOMMENDATIONS In summary, we recommend that the Regional Administrator:
• Develop a quality management plan for removal actions that
includes: (1) responsibilities for quality assurance functions;
(2) training requirements; and (3) a selection basis and tracking
system for environmental laboratories.
• Require the regional Quality Assurance Office to monitor
compliance with quality management plans, including plans for
in
-------
the Emergency Response Office and its technical contractor.
Amend the on-scene coordinators' responsibilities for QAPPs
to include the development of DQOs and consideration of
critical quality assurance activities. Also, ensure
responsibilities include QAPP approval, implementation, and
monitoring.
Require the Quality Assurance Office to review and approve all
QAPPs, including providing verbal approval at emergency
actions, if necessary.
Require on-scene coordinators to develop DQOs for all
removal actions.
Establish a minimum, mandatory training requirement for
DQOs for regional personnel whose duties involve the
collection, evaluation, or use of environmental data.
Adopt a team approach to developing DQOs using facilitators,
on-scene coordinators, quality assurance experts, statisticians,
and technical experts.
REGION 9 A draft report was provided to the Region on April 23, 1998, and
COMMENTS tne Region provided us its draft comments on July 24, 1998. We
held an exit conference with regional officials on August 12, 1998
and the Region provided us its final comments on August 21, 1998.
In its final comments, the Region said it:
...is committed to early implementation of the audit
recommendations and intends to complete substantially all
major milestones in the recommendations before the end of the
calendar year.
The Region agreed to develop a quality management plan to
address data quality at removal actions. The quality management
plan will provide for:
• A graded approach to quality assurance;
• Development of generic quality assurance project plans;
• Approval of quality assurance project plans by the Quality
Assurance Office; and,
iv
-------
• Oversight of the implementation of the quality management
plan by the Quality Assurance Office.
Also, the Region will initiate a training program to assure that all
on-scene coordinators receive appropriate training on data quality
objectives. The Region believed that these actions, together with
other ongoing efforts, should greatly enhance the management of
data quality at removal actions.
The Region said although it still had "some disagreements with
specific conclusions, we do not believe that these disagreements
detract from the overall message regarding the need to improve
data quality." The Region also said that it "firmly believes that
the five removal actions considered in preparation of the report
were appropriate and protective of human health."
The Region's response to the audit report recommendations are
summarized in Chapters 1 and 2 of our report. The Region's entire
response is at Appendix A.
-------
(This page intentionally left blank.)
VI
-------
Table of Contents
Executive
Summary
CHAPTER 1
CHAPTER 2
CHAPTER 3
Page
i
Quality System for Removal Actions Needs Improvement .. 1
Background 1
Quality Management Plan Needed 3
Data Quality Objectives Not Used to Design QAPPs 5
Critical Quality Assurance Activities Not Specified 5
Plans for Collecting Data Not Optimized 9
QAPPs Not Reviewed by Quality Assurance Office 10
Plans Not Implemented 11
Recommendations 13
Region 9 Comments 14
Data Quality Objectives Could Improve Removal Planning
and Data Collection 17
Background 17
Removal Actions Need Better Planning 18
Causes 19
Best Practices 21
Recommendations 22
Region 9 Comments 22
Background 25
Emergency Response Program 25
Audit Objective 27
Significance of Environmental Data 27
Audit Scope and Methodology 28
Prior Audit Coverage 32
-------
APPENDICES A Region 9 Response to Draft Report 33
B Removal Actions Audited 43
C Abbreviations 49
D Laws, Regulations, Directives, and Guidance 51
E Definitions of Quality Assurance Activities 53
F Hanford's Best Practice for Data Quality Objectives 55
G Report Distribution 57
-------
CHAPTER 1
Quality System for Removal Actions
Needs Improvement
BACKGROUND
Our audit of five removal
actions showed the Region did
not have a quality management
plan which adequately
documented and described the
quality system for removal
actions. Further, most of the
site-specific quality assurance
project plans (QAPPs) we
reviewed:
• Were not based on the
seven-step data quality
objectives process;
Quality System Weaknesses
• No quality management plan
for removal actions.
• DQOs not used to design
quality assurance project plans.
• Critical quality assurance
activities excluded from plans.
• Plans for collecting data not
optimized.
• Plans not reviewed by Quality
Assurance Office.
• Plans not implemented.
• Were not designed to prevent and detect inappropriate quality data;
• Did not include defensible or optimal plans for collecting data;
• Were not reviewed or approved by the Region's Quality Assurance
Office; and,
• Were not implemented or monitored.
We believe the main reason the quality management plan and QAPPs were
insufficient was because regional decision makers did not perceive data
quality as a risk to decision making.
Consequently, the Region completed three removal actions without
appropriate quality data for decision making. In addition, about 420
samples were analyzed for one removal action that were not used for the
decision indicated in the sampling plan.
According to the American national standard, Specifications and
Guidelines for Quality Systems for Environmental Data Collection and
Environmental Technology Programs, ANSI/ASQC E4-1994, a tiered
quality system is recommended that:
-------
Quality
Management Plan
Quality Assurance
Project Plan
Regional Quality
System
1. Describes the entire organization-
wide quality system in a quality
management plan, and
2. Is supported by QAPPs describing
the applicability of the quality system
to specific actions.
Two-Tiered Quality
System
Quality Management
Plan
Quality Assurance
Project Plans
EPA's quality system is prescribed in
Policy and Program Requirements for the Mandatory Agency-Wide
Quality System, EPA Order 5360.1, change 1. The order calls for
establishing quality systems that fully comply with the American national
standard.
A quality management plan should include: (1) responsibilities for
management and staff; (2) personnel qualifications and training; and (3)
documentation procedures. It should also describe when and how controls
will be applied to specific projects. EPA Headquarter's Superfund
Program has a quality management plan and each region is required to
develop a quality management plan.
"The QAPP is an important part of
the EPA Quality System, and is
required for all data collection
activities that generate data for use
by EPA."
-EPA's DQO Guidance, EPA QA/G-4
The American national
standard also requires all
work involving the generation
and use of environmental data
to be planned and
documented. Typically, this
documentation is a project-
specific QAPP. Also required
by the National Contingency
Plan, a QAPP defines in detail how specific quality assurance and quality
control activities will be applied to a project.
Both the American national standard and EPA's guidance, QA/G-5,
provide requirements and suggestions for preparing QAPPs.
Region 9 had a quality management plan. Further, the Region's
Emergency Response Office used two types of QAPPs:
1. "Generic" plans that provided guidance for most removal actions.
2. Site-specific plans (sometimes called quality assurance sampling
plans) that were required for each removal action.
The Region's Quality Assurance Office was responsible for:
-------
QUALITY
MANAGEMENT
PLAN NEEDED
Region's Plan Did
Not Address
Removal Actions
• Reviewing and approving quality assurance documents;
• Conducting audits of environmental data collection activities; and,
• Implementing Agency QA policies.
Chapter 3 provides more information on the Region's organizational
structure.
Although the Region had a quality management plan, generic QAPPs, and
a contractor quality management plan, collectively these plans did not
adequately document and describe how quality controls were applied to
removal actions. According to the American national standard,
ANSI/ASQCE4-1994:
A quality system shall be planned, established, documented,
implemented, and assessed as an integral part of a management
system for... implementing... environmental programs ...
The data quality system suffered because regional decision makers did not
believe that data quality posed a significant risk to removal actions. We
believe this lack of concern about data quality contributed to systemic
weaknesses in the quality system for removal actions. Further, the Region
had not required its programs to document their quality systems in quality
management plans.
Although Region 9 had a quality
management plan, it did not describe
a quality system for removal actions
in sufficient detail. For example, it
did not include:
• Management and staff
responsibilities for quality
assurance functions, such as
developing DQOs and monitoring
compliance with QAPPs;
• Minimum training requirements
for quality assurance;
A selection basis for environmental laboratories, such as laboratory
audits and performance evaluation samples; and,
"The quality system shall be
described in a quality
management plan
documents) or quality
manual(s) that has been
reviewed and approved for
implementation...."
- American National Standard,
ANSI/ASQC E4-1994
-------
A tracking system for monitoring laboratory performance.
Generic QAPPs
Insufficient
Contractor's
Quality
Management Plan
Not Fully
Implemented
Further, the Region's quality management plan excluded important quality
controls included in the Superfund Program's quality management plan.
Although the Emergency Response Office and its technical contractor had
generic QAPPs that set some quality assurance guidelines, these QAPPs
were not substitutes for quality management plans for two reasons:
1. They did not include many of the fundamental aspects of a quality
management system, such as responsibilities for quality assurance;
and,
2. They did not prescribe a system to control laboratory data quality that
included laboratory audits or successful analysis of performance
evaluation samples.
Also, these generic QAPPs did not apply to some types of removal actions.
For example, the Emergency Response Office's generic QAPP did not
apply to removal actions taken by the Superfund Cleanup Branch, which
handled some removal actions.
The Emergency Response Office's technical contractor had a quality
management plan but it was insufficient. The Emergency Response Office
used a technical contractor for much of its environmental sampling.
However, the contractor's quality management plan was not effective for
several reasons:
1. The contractor's plan applied only to certain removal actions that were
delegated to the technical contractor.
2. The plan was not fully
implemented. For example, we
found that the contractor had not
performed laboratory audits, an
important aspect of its quality
assurance system, although
required by its plan.
"Laboratory system audits will be
performed on subcontract
laboratories."
-Technical Contractor's Quality
Management Plan
Lack of Plan
Causes Problems
3. The Region was not checking compliance with the plan. Both the
contract and Region's quality management plan required quality assurance
program audits of the contractor. However, none had been done since the
contract became effective in December 1995.
The lack of an effective, implemented quality management plan caused
many of the problems we found with the quality system.
-------
DATA QUALITY
OBJECTIVES NOT
USED TO DESIGN
QAPPS
CRITICAL
QUALITY
ASSURANCE
ACTIVITIES NOT
SPECIFIED
Important Quality
Assurance
Activities
For example, on-scene coordinators lacked extensive training in quality
assurance. One of the reasons was that the quality management plan did
not specify quality assurance training requirements for on-scene
coordinators. Training requirements should be established to ensure on-
scene coordinators are qualified to review, approve, and monitor
implementation of QAPPs.
The lack of a quality management plan contributed to other problems, such
as QAPPs that were absent important quality assurance activities. The
following sections explain the adverse impacts caused by ineffective
QAPPs.
Our audit of five removal actions found the seven-step data quality
objectives process was not used to design QAPPs. As discussed in
Chapter 2, on-scene coordinators were not required to develop DQOs.
Setting a data quality objective is the first step in developing an effective
QAPP and is critical for effective and efficient data collection efforts.
DQOs are needed to determine the type, quantity, location, and quality of
sampling necessary for decision-making.
QAPPs were not effective in detecting inappropriate quality data because
they did not include:
• Two or more important quality assurance activities;
• Reporting and assessment criteria for validated or verified data;
• A basis for selecting laboratories; or,
• Approval by on-scene coordinators, in two instances.
As a result, data was not of known or appropriate quality for decision
making. Further, tainted data was used to evaluate a possible source of
contamination at one removal action.
Although the QAPPs we audited required data validation, an important
quality assurance activity, only two included a specification for another
important quality assurance activity:
-------
Requirement Included in QAPP
Data Reporting
and Assessment
Requirements
Criteria for
Correlating
Laboratory and
Field Data Not
Established
QAPP
Casmalia
Cruz Ranch
Dodson Brothers
Sanders Aviation
Tucson Airport
Data
Validation
•
•
•
•
•
Field
Audits
•
Performance
Evaluation
Samples
•
Magnetic
Tape
Audits
The American national standard, ANSI/ASQC E4-1994, necessitates
consideration of these activities. According to the American national
standard, the design process for data collection must consider these
activities and develop detailed specifications for them. Also, prior OIG
audits have found that a combination of quality assurance activities is
needed for an effective design because one activity alone, such as data
validation, cannot find many of the known quality problems. Appendix E
further explains how these activities can improve data quality.
Along with data validation, field audits and performance evaluation
samples should be required when data of higher quality is needed for
important decisions, such as deciding if a removal action is complete.
Magnetic tape audits should be performed if major deficiencies are found
by other methods. For tapes to be available for audit, EPA should include
a requirement for magnetic data availability for appropriate analytes in
QAPPs and corresponding contracts.
Although data validation was required
reporting requirements and criteria
for assessing validated or verified
data. They also did not include
reporting or assessment criteria for
other types of quality assurance
activities, such as performance
evaluation samples. As a result,
decisions at the Cruz Ranch and
Dodson Brothers sites were not
based on appropriate quality data.
At the Cruz Ranch site, the QAPP
did not specify criteria for
determining that screening data was
sufficient for verifying
, QAPPs did not include data
Missing Reporting
Requirements
No procedures for reporting
the results of validated data
to on-scene coordinators.
Lack of reporting
requirements for
performance evaluation and
split samples.
No criteria for determining
if data needed to be rejected.
-------
Serious Data
Problems Not
Reported
contamination was removed. To ensure contamination was removed, the
QAPP called for:
• Screening data taken by field measurements to determine that metal
contaminated soil was removed; and,
• Ten percent of the screening samples to be analyzed by a laboratory
and validated.
However, the QAPP did not establish criteria for determining if the
laboratory's analytical data sufficiently correlated with the field screening
data. As a result, no action was taken when EPA's technical contractor
found that insufficient samples were taken to correlate the field screening
data with the laboratory's analytical data. Thus, the Region concluded that
the contamination was removed although it lacked appropriate quality data
for such a decision.
At the Dodson Brothers site, EPA's technical contractor found serious
problems with the data. However, these problems were not reported to
EPA.
Dodson Brothers Site, Montclair, California
Data collected to identify an alleged waste pit was tainted. The removal
action included an objective to identify the location of an alleged waste pit.
During a week of soil-gas sampling to identify the pit, the doors to a
mobile testing laboratory were left open, allowing diesel exhaust to enter
-------
Laboratory
Selection Not
Addressed
Audits
Performance
Evaluation Samples
Coordinator
Approvals Missing
the laboratory and contaminate sample analysis equipment. Therefore, the
data was tainted.
Because reporting procedures were deficient, the on-scene coordinator was
not aware that the data was tainted. A few days after sampling was
completed, the contractor identified data problems in a data review report.
However, the QAPP did not require the contractor to promptly provide this
report to the on-scene coordinator. As a result, the on-scene coordinator
was unaware of the data problem. Consequently, the Region's decision
that the alleged waste pit did not exist was not based on appropriate
quality data.
As noted in our discussion of quality management plans, the Region's
quality management plan and generic QAPPs did not address the basis for
selecting environmental laboratories. Further, except for Dodson Brothers
site, none of the QAPPs for the sites we audited included the basis for
selecting laboratories. Without a laboratory evaluation technique, such as
audits or performance evaluation samples, the quality of data produced by
a laboratory is unknown.
Audits are necessary to ensure laboratories are technically qualified, and to
identify and prevent data problems. Laboratory audits are designed to
identify technical areas which may cause laboratories to improperly
identify and quantitate chemicals. To prevent problems, an on-site audit
should be done before any samples are analyzed.
Performance evaluation samples are another time-tested method to
evaluate a laboratory's ability to analyze a specific sample. EPA uses
performance evaluation samples extensively in its contract laboratory
program to evaluate laboratory performance.
We could not substantiate that on-scene coordinators approved QAPPs at
two of the removal actions we audited: Casmalia and Cruz Ranch. As a
result, it was unclear if these on-scene coordinators agreed the QAPPs
should have been implemented.
Evidence of approvals were missing because the Emergency Response
Office felt it was unnecessary for on-scene coordinators to indicate
approval of QAPPs by their signature.
QAPPs should be approved by on-scene coordinators for several reasons.
First, written approvals document that coordinators agree that QAPP
provisions are adequate. Second, a signed and dated QAPP provides field
and laboratory personnel assurance that they are using an approved, up-to-
date QAPP. Third, both the National Contingency Plan and EPA Order
5360.1 require EPA approval of QAPPs.
8
-------
PLANS FOR
COLLECTING
DATA NOT
OPTIMIZED
Samples Not Used
for Intended
Purpose
Three of the five sites we audited had plans for collecting data that were
not optimal or defensible. As a result, samples were analyzed at one site
that were not used for the decision indicated in the sampling plan. Also,
sample designs did not support defensible decision making at two other
sites.
Sampling plans document the design for collecting samples and explain
the basis for the design. These plans, components of QAPPs, provide
specific details about the type, quantity, and location of samples to be
analyzed during a removal action. The rationale for taking the samples
should be included so that decisions are defensible and appropriately
documented.
The Region did not use the data from about 420 effluent samples at the
Casmalia site for the decision indicated in the sampling plan. As part of
its removal action, the Region installed a treatment system for
groundwater contaminated by a pesticides and solvents landfill.
Treatment System, Casmalia Resources
Although the system effluent was sampled periodically, sample data was
not used for the decision described in the sampling plan: determination of
the suitability of the leachate for discharge. Although the first 16 samples
were used to determine if the leachate should be discharged, we estimate
the Region analyzed about another 420 samples over the four years that
-------
Sampling Plans
Not Defensible
QAPPS NOT
REVIEWED BY
QUALITY
ASSURANCE
OFFICE
were not used for this decision. The Region could not provide us with the
cost of the sampling.
Sampling plans did not ensure that data needed for defensible decision-
making was collected. We found:
• The sampling plan for the Dodson Brothers removal action was
incomplete for drawing conclusions about an alleged waste pit as a
source of contamination.
• The sampling plan for the Sanders Aviation removal action was
incomplete for drawing a conclusion on the remaining risks from DDT
or its residues.
In both of these cases, we believe both use of the DQO process and
approval of plans by the Quality Assurance Office would have likely
eliminated sampling design problems.
One of the reasons that QAPPs
had design deficiencies was
because the Region's Quality
Assurance Office did not review
or approve them. Except for the
Sanders Aviation QAPP, none of
the project specific QAPPs we
audited were reviewed or
approved by the Quality
Assurance Office.
Reasons for Quality Assurance
Review and Approval of QAPPs
Regional quality management plan
requires the Quality Assurance
Office's review and approval.
Regional experts should be
included in all quality assurance
planning.
Early Quality Assurance Office
involvement would expedite the
QAPP review process.
On-scene coordinators are
responsible for approving QAPPs.
As part of this process, coordinators should submit QAPPs to the Region's
Quality Assurance Office. This office is responsible for reviewing quality
assurance documents and has expertise in chemistry, quality assurance,
and sample design.
On-scene coordinators told us they did not submit QAPPs to the Quality
Assurance Office for review and approval because:
• Removal actions were done quickly and coordinators could not afford
the two or three weeks the Quality Assurance Office needed to review
and approve QAPPs; and,
• The Quality Assurance Office's suggested changes did not "add any
value" to QAPPs.
10
-------
Perceived Lack of
Time
Perceived Lack of
Value Added
PLANS NOT
IMPLEMENTED
On-scene coordinators and the Quality Assurance Office need to develop a
time-sensitive, team approach to combine their expertise to develop
defensible designs for collecting data. These designs are manifested in
QAPPs.
On-scene coordinators perceived that they did not have enough time for
the Quality Assurance Office to review and approve their QAPPs. The
main reason cited was that the "time critical" nature of removal actions did
not allow time for normal reviews and approvals.
We found that half of the
Region's removal actions in
process between October 1,
1995 and March 31, 1997 took
more than six months to
complete. The five removal
actions audited took an average
of 26 months to complete.
These times ranged from
1 month to more than 4 years.
Months Taken to Complete 30 Removals
October 1,1995 to March 31,1997
Over 12
Months
7 to 12
Months
Oto1
Month
2 to 6
Months
Thus, we believe it is reasonable
to allow the Quality Assurance Office time to review QAPPs. For some
extremely short duration removal actions, an expedited procedure may be
necessary to obtain the approval of the Quality Assurance Office.
On-scene coordinators told us the Quality Assurance Office did not
significantly improve QAPPs, considering the time investment. However,
we found that Quality Assurance Office reviews did identify significant
weaknesses in QAPPs. For example, the Quality Assurance Office found
that the QAPP for the Sanders Aviation removal action did not include a
cleanup level for DDT. The Region should develop an expedited QAPP
review process that identifies and resolves significant data quality issues.
Although one or more effective quality assurance activities were included
in QAPPs, often they were not performed. This was the case with three of
the five sites we audited:
11
-------
QAPP Requirements Not Implemented
Coordinators Did
Not Monitor
QAPPs
Adverse Impacts
Insufficient Data
Removal
Action
Dodson
Brothers
Sanders
Aviation
Tucson
Airport
Data
Validation
•
•
•
Field
Audit
•
i
1
Laboratory
Audit
•
i
i
PE
Sample
i
i
•
1- Not included as a QAPP requirement.
One of the main reasons QAPPs were not implemented was because on-
scene coordinators did not monitor QAPPs and, consequently, were not
aware that QA activities were not performed. As noted previously, the
Region did not have a quality management plan that identified on-scene
coordinator responsibilities for data quality, such as monitoring QAPP
implementation. These responsibilities need to be identified and carried
out.
For two of the removal actions audited, we concluded that:
• A decision at the Sanders Aviation removal action was not based on
appropriate quality data and,
• One objective of the Dodson Brothers removal action was not
achieved.
Data generated for the Sanders Aviation site at a mobile laboratory was
not validated. Therefore, the data was not of appropriate quality for
determining the cleanup was complete. This removal action cost about
$2.8 million.
"Ten percent of the samples must
be confirmed with QA Definitive
data."
-Sanders Aviation QAPP
Contrary to the requirements of its
QAPP, the Region used test results
from a mobile laboratory to confirm
that toxaphene was cleaned up. The
sampling plan required the regional
laboratory to analyze 10 percent of
the final samples and the results be validated. However, the results of the
laboratory analyses were not validated or provided to the on-scene
coordinator. This was because the on-scene coordinator was not
monitoring compliance with the QAPP. Consequently, the coordinator did
not have sufficient data to determine whether toxaphene-contaminated soil
exceeding the cleanup level was left in place.
12
-------
Objective Not
Achieved
Sanders Aviation Site, Tempe, Arizona
At the Dodson Brothers site, a QAPP for the removal action included an
objective to evaluate the threat to groundwater, but the objective was not
done. Although the Emergency Response Office's generic QAPP required
a significant QAPP deviation to be documented and explained, the
deviation was not explained. Further, the Region did not identify the
impact the change had on meeting the DQOs, as required by the generic
QAPP. As a result, the Region did not have sufficient documentation to
explain why a threat to groundwater, the basis for the removal, was not
evaluated.
RECOMMENDATIONS We recommend that the Regional Administrator:
1. Develop a quality management plan for removal actions that includes:
• Management and staff responsibilities for quality assurance
functions, such as development of DQOs and monitoring
compliance with QAPPs.
• Minimum training requirements for quality assurance.
• Selection basis for environmental laboratories, including
requirements for laboratory audits and performance evaluation
samples.
1 i
-------
• A tracking system for monitoring laboratory performance.
• Significant quality controls included in the Superfund Program
quality management plan.
2. Require the Quality Assurance Office to monitor compliance with
quality management plans including plans for the Emergency
Response Office and the technical contractor.
3. Ensure on-scene coordinator responsibilities for QAPPs include:
• Development of DQOs.
• Consideration of critical quality assurance activities and
procedures, including data validation, field audits, performance
evaluation samples, magnetic tape availability for audits,
laboratory audits, data reporting requirements, and criteria for
assessing limitations on data use.
• Formal approval of QAPPs.
• Implementation of QAPPs.
4. Have the Emergency Response Office and Quality Assurance Office
develop a time-sensitive, team approach for developing defensible
QAPPs. Require the Quality Assurance Office to review and approve
all QAPPs, including providing verbal approval at emergency response
actions if necessary.
REGION 9 The Region agreed with the recommendations and stated it would:
COMMENTS
• Develop a program-specific quality management plan for the
Emergency Response Office.
• Amend the on-scene coordinators' responsibilities to include the
development of data quality objectives and consideration of critical
quality assurance activities.
The Emergency Response Office will involve the Quality Assurance
Office in "scoping" for quality assurance activities. The Quality
Assurance Office will also have the responsibility to review and approve
all QAPPs.
14
-------
In its comments to the draft report, the Region stated that although the
audit report has valid areas for improvement:
...the Region does not believe these areas invalidate the
appropriateness of the [on-scene coordinators'] judgements under
the circumstances present, particularly when one takes into
account the specific role of removals in achieving significant risk
reduction. Similarly, none of the response actions taken was
compromised by any of the alleged deficiencies in the DQO
process...
The Region also noted that differences between the remedial and removal
programs should be taken into account in assessing the DQO processes
followed in the two programs. The Region advised that sampling data
may be a relatively small part of the decision matrix for removal actions
and serve the purpose of filling in details, rather than defining the basic
parameters of a decision. The most quality-critical needs at removal
actions typically arise at the end of the project when cleanup confirmation
is desired and expected concentrations are lower. This data collection
point is well after the most important decisions have been made.
15
-------
(This page intentionally left blank.)
16
-------
CHAPTER 2
Data Quality Objectives Could Improve
Removal Planning and Data Collection
BACKGROUND
Incomplete DQO Process
Casmalia Resources
Cruz Ranch
Dodson Brothers
Sanders Aviation
Tucson Airport
The Region undertook removal actions
and corresponding data collection
efforts without using the seven-step
DQO process, EPA's systematic
planning process.
The Region did not use the complete
DQO process at the five removal actions we audited because on-scene
coordinators were not required to and they were generally unfamiliar
with the DQO process. Also, the Region did not have a system in place
to support the process.
As a result, the Region completed these removal actions, costing more
than $20 million, without documenting important decision criteria or
alternatives.
DQOs are precise statements describing the
objective of a data collection effort. The
DQO process is a systematic, scientific
method to establish data quality criteria and
performance specifications for decision
making. The DQO process was developed by
EPA to:
• Help define specific questions that an
environmental project is intended to
answer;
• Identify the decisions that will be made
when using the resulting data;
• Define the allowable risk of decision
errors in specific and quantifiable terms;
and,
• Optimize the design of data collection.
Underlying
Principles of
DQOs
• All collected data
have error.
• Nobody can afford
absolute certainty.
• The DQO process
defines tolerable
error rates.
• Absent DQOs, deci-
sions are
uninformed.
17
-------
REMOVAL
ACTIONS NEED
BETTER
PLANNING
The DQO process provides two primary benefits:
1. Better decisions, because they are based on the scientific method
and decision error is reduced.
2. More cost effective data collection efforts, because managers
focus on the quantity and quality of data needed for decisions.
EPA's Quality Assurance and
Superfund program guidance provide
detailed instructions for implementing
the DQO process. EPA's 1994
Guidance for the Data Quality
Objectives Process, QA/G-4, provides
a seven-step process to help project
managers collect appropriate data.
(See box, right.) Also, OSWER
publication 9355.9-01, Data Quality
Objectives Process for Superfund,
provides detailed guidance for
developing DQOs.
DQOs are an essential element of
EPA's quality system. As shown in
The EPA Quality System, QA/G-0,
DQOs are the foundation of EPA's
quality assurance planning for projects.
Further, the quality management plan
for Superfund requires the DQO
process for all projects involving
environmental data collection.
Our audit of five removal actions
found that the DQO process was not
followed. As a result, more than $20
million was spent on these removal
actions without adequately
documenting decision criteria or
performance specifications for decision making, as shown in the
following table:
Data Quality Objectives
Process
State the Problem to
be Solved
L
V
Identify the Decision to
be Made
T
Identify Inputs to the
Decision
T
Define the Study
Boundaries
T
Develop a Decision Rule
1
r
Specify Tolerable Limits
on Decision Errors
1
r
Optimize the Design for
Obtaining Data
18
-------
Missing Decision Criteria
At Five Removal Actions
Removal
Action
Casmalia
Cruz
Ranch
Dodson
Sanders
Tucson
Missing Decision
Criteria
Decision rules, action
alternatives, and
performance specifications
not sufficiently documented.
Decision criteria for
correlating data not defined.
Decision criteria and error
rate for an alleged source of
contamination not defined.
Decision rule for DDT, a
contaminant, not defined.
Decision alternatives using
risk-based cleanup criteria
not fully explored.
Effect on Removal
Actions and Data
Missing performance
criteria to evaluate action
completion. Some
unneeded data collected.
Insufficient data to support
conclusion that
contamination removed.
Unquantified risk that a
source of groundwater
contamination may exist.
No confirmatory samples
to show DDT risks were
abated.
Soil unnecessarily
removed. Associated
sampling unneeded.
The cost of each removal action is identified in Appendix B.
CAUSES
DQOs Not
Considered
Mandatory
DQOs were not developed because on-scene coordinators:
• Were not required to develop them.
• Did not have sufficient training or experience developing DQOs.
Why DQOs Were Not Used
Also, one coordinator thought the
DQO process was not practical at
removal actions. Moreover, the
Region had not developed a
system to support DQO
development.
On-scene coordinators did not
believe they were required to
develop DQOs. However, the
Superfund quality management
plan, the Region's quality management plan, and the technical
contractor's quality assurance project plan required the DQO process for
DQOs were not considered
mandatory.
Lack of DQO training and
experience.
Perception that DQOs were
not practical.
Process to support DQOs not
in place.
19
-------
Training and
Experience
Insufficient
DQOs Are Practical
all sites, as part of project planning. Further, EPA's quality system, as
described in QA/G-0, uses DQOs as the foundation of its quality system
at the project level.
The DQO process should be required for removal actions to facilitate
sound decision making. As discussed below, an abbreviated DQO
process could be used for "classic" emergencies.
On-scene coordinators did not have sufficient training and experience
with DQOs.
Only two of the five on-scene coordinators whose actions we audited
recalled attending a specific training course on the data quality objective
process. Only one had used the process. We also found only one of the
Region's ten on-scene coordinators attended either of the two DQO
courses sponsored by the Quality Assurance Office since December
1994.
The Region needs to establish minimum training requirements on DQOs
for project managers and quality assurance experts. On-scene
coordinators need to understand the DQO process before it can be
implemented.
One on-scene coordinator felt the DQO process was too lengthy for
time-critical removal actions. However, we found:
• Over half the removal actions took over 6 months to complete, as
detailed in Chapter 1, affording at least a few days to develop DQOs.
• A "graded approach" to DQOs could be used to tailor the process for
removal actions completed in less than 6 months.
DQO guidance recognizes a "graded approach" that bases the level of
application according to the intended use and the degree of confidence
needed in the results. While the depth and detail of DQO development
vary, the process would benefit all removal actions by improving
documentation of decisions and activities, fostering communication
among participants, and clarifying vague objectives.
The Region may also wish to consider developing "pro-forma" DQOs
for classic emergencies or for similar types of recurring projects, such as
illegal drug laboratories.
20
-------
Support Process
Needed
Team Approach
Facilitators
BEST PRACTICES
Changes Needed to
Support DQO Process
• Require DQOs
• Set training requirements
• Use a team approach
• Designate facilitators
The Region did not have a process in
place to support DQO development.
Successful applications of DQOs have
involved a team approach using
facilitators to guide the process.
The Region needed to implement a
team approach for developing DQOs ~ ~~~ r™-—™™^-™
that included on-scene coordinators,
the Quality Assurance Office, statisticians, and technical experts. We
found these experts usually were not consulted during removal action
decision-making. According to quality assurance experts, a problem will
have the greatest chance of being solved when a multi-disciplinary team
of technical experts can help to recognize all of the important facets of
the problem and ensure that complex issues are described accurately.
The Region needed to designate facilitators to guide decision makers
through the process. As noted in the following section, one of the
success factors in implementing the DQO process was the use of
facilitators to keep the process moving and on track.
The U.S. Department of Energy seems to have been particularly
successful implementing the DQO process. It should be noted that
Energy has required the use of the DQO process at its environmental
projects and operations.
The Department of Energy sponsors a DQO Internet "web" site that
explains the DQO process, provides case studies of lessons learned and
cost savings, and identifies DQO resources. The address is
http://etd.pnl.gov:2080/DQO/.
Our audit of Laboratory Data Quality at Federal Facility Superfund
Sites, issued in March 1997, found that the Hanford Nuclear Reservation
had developed an effective DQO implementation procedure. This
procedure, shown at Appendix F, involves key decision makers in the
development of objectives. The Region should consider implementing
many aspects of this procedure.
A key part of Hanford's DQO process was the use of a facilitator. The
facilitator can assist by fostering communication among planning team
members and adding objectivity to the decision making process. The
facilitator should have a broad range of technical and regulatory
expertise and experience in making focused decisions.
21
-------
RECOMMENDATIONS We recommend that the Regional Administrator:
1. Require on-scene coordinators to develop DQOs for all removal
actions.
2. Establish a minimum mandatory training requirement for DQOs for
all regional personnel whose duties involve the collection,
evaluation, or use of environmental data.
3. Require on-scene coordinators to attend DQO training.
4. Use a graded or pro-forma approach to develop DQOs depending
upon the scope and complexity of the project.
5. Use a team approach to develop DQOs. The team should include
QA specialists, samplers, chemists, project managers, risk assessors,
lexicologists, data users and statisticians.
6. Designate a DQO facilitator to assist and coordinate team members
through the DQO process.
REGION 9 The Region agreed with the recommendations and stated it would:
COMMENTS
• Require on-scene coordinators to develop DQOs for all removal
actions;
• Establish a minimum DQO training requirement;
• Require on-scene coordinators to attend DQO training;
• Develop site-specific data quality objectives and use generic DQOs
in emergency situations;
• Use a team approach to develop DQOs involving on-scene
coordinators and the Quality Assurance Office; and,
• Use a trained facilitator to assist in DQO development.
Most of these requirements will be included in the Emergency Response
Office's quality management plan, scheduled to be developed by
December 31, 1998. All on-scene coordinators will be trained in DQOs
by March 31, 1999.
22
-------
As noted in the Region's comments to Chapter 1, the Region believed
none of the response actions were compromised by any deficiencies in
the DQO process. It also stated that differences between the remedial
and removal programs should be taken into account in assessing the
DQO processes followed in the two programs.
23
-------
(This page intentionally left blank.)
24
-------
CHAPTER 3
Background
This section provides information on EPA's Emergency Response
Program, along with the audit objective, significance of the audit area,
scope, and methodology. It also discusses prior audit coverage.
EMERGENCY
RESPONSE
PROGRAM
What Rules Apply?
EPA's Emergency Response Program responds to threats posed by
the sudden or unexpected releases of hazardous substances. This
program is carried out by its ten regional offices in close cooperation
with a network of federal, state, and local government agencies.
Region 9's Emergency
Response Office manages
the majority of EPA's
responses, called removal
actions, throughout
California, Nevada, Arizona,
Hawaii, and the Pacific
Islands. In fiscal 1997, the
Office handled a wide range
of actions, including
cleaning up vinyl chloride
contamination in West
Oakland. The Office had
ten on-scene coordinators
and used two contractors for
technical support and
cleanup services.
The Site Cleanup Branch
and the Federal Facilities Branch, also part of the Superfund Division,
manage some removal actions at National Priority List sites.
The Quality Assurance Office sets regional quality assurance policies,
reviews quality assurance documents, and advises program managers
on quality assurance matters.
Removal actions are governed by Superfund legislation, the National
Contingency Plan, and EPA procedures.
25
-------
Superfund Laws
The Emergency Response Program is regulated by Superfund laws:
the Comprehensive Environmental Response, Compensation and
Liability Act (CERCLA) of 1980 and the Superfund Amendments and
Reauthorization Act of 1986. These laws gave EPA the ability to
respond to uncontrolled releases of hazardous substances, provided for
liability of parties responsible for these releases, and established a trust
fund to provide for a cleanup when no responsible party could be
identified.
National Contingency
Plan
The National Oil and Hazardous Substances Pollution Contingency
Plan, commonly called the National Contingency Plan, is the Federal
government's blueprint for responding to both oil spills and hazardous
substance releases.
Once EPA learns of a possible hazardous substance release, it
investigates the site, evaluates the threat, and determines the best
course of action. First, EPA designates an on-scene coordinator to
evaluate the incident and determine the appropriate response agency.
If the on-scene coordinator determines that EPA will take the lead in
responding to the incident, he
or she will evaluate the
urgency of the situation to
determine the appropriate
level of removal.
Levels of Removal Actions
• "Classic" Emergencies. On-
site actions must start within
minutes or hours of the
determination that an action is
needed.
• Time-Critical Actions. Less
than six months is available
before on-site actions must start.
• Non-Time-Critical Actions.
More than six months is available
before on-site actions must begin.
All removal actions are not
equally urgent. For example,
situations involving
imminent, catastrophic
contamination of a reservoir
may require expeditious
attention, while situations
involving abandoned waste
drums may not. Based on the
National Contingency Plan,
there are three categories of
removal actions:
emergencies, time-critical
actions, and non-time critical actions. (See box.)
All removal actions are not intended to be large-scale, long-term
cleanups. The National Contingency Plan set a $2 million dollar limit
and 12-month time limit on removal actions, unless there are
extenuating circumstances.
26
-------
EPA Procedures
AUDIT OBJECTIVE
EPA has developed a ten-volume series of guidance documents
collectively called Superfund Removal Procedures. These procedures
include preparation of an action memorandum, which discusses the
removal actions taken, and procedures for ensuring underlying
environmental data is sound for decision-making. EPA Order 5360.1
holds regional managers responsible for ensuring quality systems
produce data of adequate quality for environmental decisions.
EPA has listed its generic quality assurance references on the Internet.
The address is http://www.epa.gov/regionlO/www/offices/oea/
qaindex.htm.
The objective of the audit was to determine if Region 9 had sufficient
procedures in place to ensure that environmental data was of known
and acceptable quality for Superfund removal actions.
SIGNIFICANCE OF
ENVIRONMENTAL
DATA
Laboratories Create
Serious Data
Problems
We performed this audit because of known risks to the quality of
environmental data. Region 9's Superfund program has experienced
serious problems with environmental data quality that are likely to
continue. Risks to environmental data are significant because data is
the basis for EPA's decision making and enforcement actions.
According to the American national standard, ANSI/ASQC E4-1994,
environmental data is any measurement or information that describes
environmental processes or conditions, or the performance of
environmental technology. Data is typically collected by sampling and
analyzing water, soil, or air for possible contaminants.
Since 1995, at least five environmental laboratories have had serious,
systemic problems with environmental data quality, including falsified
data. These laboratories were used by at least 16 Superfund sites in
Region 9. For example:
• Extensive fraud was found at one environmental laboratory,
resulting in $5 million of lost data, resampling costs, and
associated expenses. The laboratory was used at nine Superfund
sites in Region 9.
• Another environmental laboratory, used at four regional Superfund
sites, was suspended in 1996 for taking improper shortcuts and
falsifying test results. The laboratory received more than $5
million in government contracts since 1990.
27
-------
Risks Are Likely to
Continue
Competition in
Laboratory Industry
Popularity of Small
Laboratories
AUDIT SCOPE AND
METHODOLOGY
Scope
Risks to the quality of data are likely to continue for at least two
reasons:
1. Intense competition in the laboratory industry; and,
2. Increasing popularity of small environmental laboratories.
Intense competition in the environmental laboratory industry is surely a
contributing factor to fraud, manipulation, and quality assurance
problems.
How does the intense competition impact the quality of environmental
data? Some laboratories may manipulate data to keep clients happy.
For example, a laboratory may report their client meets EPA
requirements, thereby helping their client avoid fines, compliance
measures, and more frequent testing.
Another possible motivation for manipulating data: cutting costs. A
failed or improperly performed test is sometimes done at the
laboratory's expense.
The increasing popularity of small environmental laboratories have
increased the risks to environmental data. Small laboratories typically
lack important quality controls, such as an independent quality
assurance function; qualified, competent analysts; instruments that are
well-maintained; and standard operating procedures.
This section describes the audit scope and methodology, along with
prior audit coverage.
We performed our audit in accordance with Government Auditing
Standards issued by the Comptroller General. Our field work was
conducted from May to October 1997. The audit included
management procedures in effect as of March 31, 1997.
This audit resulted from a pilot review of data quality at the Tucson
International Airport Superfund site, managed by the Region's
Superfund Site Cleanup Branch. This regional site included a removal
action. We selected four additional removal actions from those
managed by the Emergency Response Office between October 1, 1995
and March 31, 1997, as shown below:
28
-------
Methodology
Site
Casmalia
Resources
Cruz
Ranch
Dodson
Brothers
Sanders
Aviation
Tucson
Airport
Reason Selected
This action was the largest within our audit scope.
The Emergency Response Office lead the removal
action and Superfund monies paid for the action. The
Region's technical contractor was responsible for
much of the data.
The Emergency Response Office lead the removal
action and Superfund monies paid for the cleanup.
The Region's technical contractor was responsible for
data.
This removal action was lead, in part, by responsible
parties. These parties funded the cleanup and
contracted for data collection and analyses.
The Emergency Response Office asked us to include
this site in our audit. Much of the environmental data
was analyzed by the Region's laboratory.
The Region's Superfund Site Cleanup Branch,
responsible for National Priority List sites, oversaw
this action. Responsible parties, under EPA order,
collected and analyzed data.
We selected these actions in order to evaluate management controls
over environmental data in a variety of funding and contracting
situations. We believe these five sites are typical of the removal
actions managed by the Region. Appendix B describes these sites in
more detail.
From October 1, 1995 to March 31, 1997, Region 9 managed 35
removal actions, excluding those at Federal facilities.
This section describes:
• Our assessment of management controls over data quality at
removal actions;
• Region 9's and EPA's 1997 Integrity Act Reports, as they relate to
management controls over removal actions;
• Analysis techniques we used to assess the adequacy of quality
assurance project plans and data quality objectives;
29
-------
Management Controls
Assessment
Federal Managers'
Financial Integrity Act
• Evidence considered in reaching our conclusions and
documentation limitations; and,
• Our evaluation of compliance with the laws and regulations.
As part of our evaluation of management controls, we performed a risk
assessment. This risk assessment identified the potential threats to data
quality and the management controls to address these threats.
Based on this risk
assessment and prior audit
work, we identified critical
management controls (see
the box, this page).
The management control
weaknesses we found are
described in this report,
along with
recommendations for
corrective action. These
weaknesses were a
significant contributing
cause to data quality
problems at the sites we
audited.
In assessing management
control weaknesses, we also
reviewed:
• EPA's 1997 Integrity
Act Report to the
President and Congress;
and,
• Region 9's Fiscal 1997
Integrity Act Report,
issued in October 1997.
In its 1997 Integrity Act
Report, EPA reported that it
corrected its material weakness with environmental data quality, first
reported in 1992. However, EPA's Senior Leadership Council was
concerned the Agency would continue to have problems with data
Management Controls Over
Data Quality
1. Oversight provided by on-scene
coordinators.
2. National standards and EPA
guidance for data quality.
3. Data quality objectives.
4. Regional quality assurance
experts.
5. Quality management plans for
the Superfund Program, the
Region, and the Superfund
technical contractor.
6. Quality assurance project plans
designed for the Emergency
Response Office and its technical
contractor.
7. Site-specific quality assurance
plans.
8. Training and experience
received by on-scene coordinators.
-EPA OIG
30
-------
Analysis Techniques
Compliance with
Laws, Regulations,
and Guidance
Evidence Considered
quality across programs. While recommending that the Administrator
report this weakness as corrected, it advised that:
"ORD [Office of Research and Development], OARM [Office
of Administration and Resource Management] and Office of
Policy, Planning and Evaluation (OPPE) will work with the
Office of the Chief Financial Officer (OCFO) and other EPA
offices and stakeholder
organizations to ensure
appropriate attention to,
and accountability for, the
quality of EPA's
"...the SLC [EPA Senior
Leadership Council] is concerned
that the Agency may continue to
have problems in oversight and
control of data quality across
programs."
-Memorandum from the SLC to the
Administrator, December 1997
environmental data."
Region 9's Integrity Act Report
did not include a management
control weakness for data
quality at removal actions.
However, the Region identified a vulnerability in quality assurance
procedures used by its contractors in subcontracting for laboratory
analyses.
In evaluating procedures over data quality, we analyzed data quality
objectives and quality assurance project plans.
To analyze their adequacy, we considered EPA and regional guidance,
the Superfund quality management plan, and the national standard for
environmental data quality. We also compared quality assurance
requirements for removal actions to requirements found effective at
other Superfund sites.
In performing the audit, we tested compliance with the National
Contingency Plan and EPA regulations and guidance, including EPA
Order 5360.1; QA/G-5, EPA Guidance for Quality Assurance Project
Plans; and EPA/540/G-90/004, Sampling QA/QC Plan and Data
Validation Procedures. Our audit of five removal actions found
instances of noncompliance, as detailed in Chapters 1 and 2.
We also reviewed quality management plans, quality assurance project
plans, action memoranda, other site-specific records (such as
enforcement orders), and training records provided to us by on-scene
coordinators. We considered the Quality Assurance Office's records
relative to training and quality assurance plan review and approval.
We interviewed officials in Region 9's Superfund Division and Office
of Policy and Management. These interviews included office
31
-------
Data Limitations
PRIOR AUDIT
COVERAGE
directors, on-scene coordinators, project officers, contracting officer,
and quality assurance chemists. We interviewed the Region's
technical contractor and contractors for responsible parties.
The Emergency Response Office informed us that documentation may
be lacking in some areas relative to environmental data quality. We
believe this lack of documentation may have impacted our ability to
accurately assess training provided to on-scene coordinators and to
fully evaluate the Region's rationale for decision making at certain
sites. When documentation was inconclusive, we interviewed regional
officials and considered verbal information.
Related EPA OIG audits of environmental data quality have found
problems with EPA's oversight of the quality of data used for
Superfund decision making. These audits include:
• Region 8 Needs to Further Strengthen Its Superfund Field
Sampling Quality Assurance Controls Activities, March 25, 1998;
• Laboratory Data Quality at Federal Facility Superfund Sites,
March 20, 1997;
• Special Review of EPA Region 9 Data Quality Oversight at the
Aerojet Superfund Site, March 28, 1996; and,
• Environmental Data Quality at DOD Superfund Sites in Region 9,
September 26, 1995.
32
-------
APPENDIX A
Region 9 Response to Draft Report
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
REGION 9
75 Hawthorne Street
San Francisco
AUG 2 1 1998
MEMORANDUM
SUBJECT: Environmental Data Quality at Superfund Removal Actions in EPA Region 9
Draft Audit Report No. E1SFF7-09-0058
FROM: Nora L. McGce, Assistant Regional Administrator
for Policy and Management (PMD-1)
TO: Truman R. Beeler
Divisional Inspector General for Audits
Office of the Inspector General for Audit (IGA-1)
Attached arc Region 9's comments on the Draft Audit No. E1SFF7-09-0058, titled Environmental Data
Quality at Superfund Removal Actions in EPA Region 9. This response provides a substantive position on the
recommendations and our corrective action plans.
The Region is committed to early implementation of the audit recommendations and intends to complete
substantially all major milestones in the recommendations before the end of the calendar year. Among the steps the
Region will take are the following:
• The Emergency Response Office (ERO) and Quality Assurance Office (QAO) will develop a
Quality Management Plan (QMP) to address data quality at removal actions;
• The QMP will provide for a graded approach to quality assurance, including development of
generic Quality Assurance Project Plans (QAPPs);
• The QMP will provide for formal QAO oversight of ERO's implementation of and compliance
with the QMP, including approval of all QAPPs (both generic and site-specific); and
• The Region will initiate a training program to assure that all On-Scene Coordinators receive
appropriate training in data quality objectives.
These actions, together with other ongoing efforts, should greatly enhance the management of data quality
at Region 9 removal actions. QAO is currently reviewing procedures used by ERO's contractor, Ecology &
Environment, and will produce an evaluation in the fourth quarter of FY98. This evaluation will include issues such
as laboratory selection, evaluation and oversight.
Region 9 appreciates the efforts of OIG staff in working with the Region to assure that the final report
accurately characterizes the removal actions which were examined in the audit. Although we still have some
disagreements with specific conclusions, we do not believe that these disagreements detract from the overall message
regarding the need to improve data quality. Consequently, we have elected not to include extensive discussions of
specific disagreements on specific cases, but to direct our energies to implementation of the recommendations.
33
-------
Although the draft report has identified specific instances in which data quality or data quality documentation
could have been improved, Region 9 firmly believes that the five removal actions considered in preparation of the report
were appropriate and protective of human health.
Region 9 believes that the conduct of this audit has also been useful in helping us to identify some ways to
improve the manner in which the Region interacts with OIG in the early planning of OIG audits. Early involvement of
Regional management can serve a useful function in avoiding the types of miscommunications that appear to have
occurred in the early development of prior drafts of the report.
Should you require a more detailed outline of the Region's plans for implementation of the recommendations,
please let us know.
Should you or your staff have any concerns or comments, please contact Rich Hennecke, Regional Audit
Follow-up Coordinator at 4-1630. Thank you for your cooperation in this project.
Attachments
34
-------
I. COMMENTS ON REPORT RECOMMENDATIONS
A. Introduction
EPA Region 9 concurs with all of the draft report's recommendations. The recommendations appear to be fully
consistent with an improved approach to data quality management in the Emergency Response Office and consistent
with the graded approach to quality assurance.
The graded approach to quality assurance of environmental data collection activities is encouraged by Agency
policies. This approach determines a logical amount of QA/QC activities for each type of site characterization/cleanup.
A removal action generally is a short term action associated with a sense of urgency with the primary purpose of risk
reduction while a remedial action generally is a much longer term effort to assess and address virtually all environmental
risks at a site. In addition, a removal action is often associated with a much more focused data collection program both
in terms of the number of samples as well as the number of analytes in comparison with a remedial action. Because a
removal action is usually intended to reduce immediate risk, it could be followed up by a longer term assessment. The
graded-approach would call for a streamlined QA/QC program for removal and a more elaborate program for remedial
actions.
If one takes the graded-approach even further, a quick emergency response action should have an even more
streamlined QA/QC procedure than a non-time-critical (NTC) removal action. Even within the same category, for
example NTC removal actions, the level of QA/QC requirements would depend on the Data Quality Objectives (DQOs)
for individual sites.
There is a lack of Agency-wide guidance on how to apply the graded approach to different types of Superfund
actions, particularly removal actions. This lack of guidance gives rise to the vagueness of how QA/QC requirements
apply to removal programs. This vagueness has also caused inconsistency in the application of QA/QC requirements
and therefore is potentially a problem larger in scope than just Region 9.
Region 9 agrees that QA documentation is absolutely essential. The extent of QA/QC program is defined by
national guidance, the infrastructure, frame work, and program accountability. In addition, each QAPP and each
sampling and analysis plan needs to define the data quality needed and associated QA/QC activities to ensure that
adequate data quality is achieved. The needed data quality is usually based on site-specific DQOs. However, in
situations where there is insufficient time to develop site-specific DQOs, a set of generic DQOs could be used for the
purpose of risk reduction. When generic DQOs are used, rationales for using the generic rather than the site-specific
DQOs should be documented.
Region 9 believes that the QA/QC program and QA/QC plan requirements applicable to the Emergency
Response Office (ERO) should be more streamlined and more suited for its mission.
Region 9 believes most of the draft audit's recommendations can be met through increased interaction and
partnership between the QAO and ERO. Interaction between the QAO and ERO should begin as early as possible in the
removal project planning phase. This interaction will add significant value for removal actions for the following
reasons:
a) Input can be provided in a timely manner by the QA Office during scoping meetings. A QA specialist can
work closely with a START contractor to quickly plan and document the necessary QA/QC activities before a
removal action. In the event of an emergency response where there is no time for planning, it is important for
the QA office to be involved from the inception. Even under an emergency or time-critical removal action, an
experienced QA professional can provide on-the-spot input and existing standard operating procedures (SOPs)
for field sampling and/or analytical methods before the emergency response occurs. Although the Agency
must accept less rigorous QA/QC under emergency situations, timely input from the QAO can add substantial
value to the quality and the defensibility of the data used in removal actions.
35
-------
b) The QA specialist could be available by phone during an emergency response action should any questions
arise from the field. The QA specialist can provide guidance to the START contractor to complete any QA
documentation and data review after the response action.
c) The QA specialist acts as a team player from the beginning of a removal project rather than being limited to
involvement in critiquing and reviewing an end product. The early involvement of the QA specialist can
actually save time for the OSC by lessening his/her workload for QA/QC review and approval.
B. Comments on Specific Recommendations
See attached table for a summary of Region 9 responses and timelines for corrective action to OIG
recommendations.
1-1 Develop a quality management plan for response [removal] actions that includes: (I) responsibilities for
quality assurance functions; (2) training requirements; and (3) a selection basis and tracking system for environmental
laboratories.
EPA Region 9 Response
EPA Region 9 agrees with this recommendation. Region 9 will develop a QMP for the Emergency Response
Office in keeping with the Regional QMP. This QMP will provide the basis for quality assurance activities for ERO and
the START contractor. The QMP will be a joint effort by the Emergency Response Office and the Quality Assurance
Office and will be approved by both offices. The QMP should also be available for removal actions conducted by the
Site Cleanup Branch.
Sampling and analysis plans (SAPs) are sometimes prepared in place of QAPPs. A SAP can contain elements
of a sample plan and a QAPP. Thus, monitoring compliance with SAPs should also be included as part of QA
functions, including review and approval of SAPs.
Region 9 agrees that some minimum QA/QC training requirements should be defined for OSCs. Training
requirements should also be defined for START contract personnel participating in the planning, collection and/or use of
environmental data. QA/QC training should include specialized curriculum suitable for removal actions.
The Region-wide QMP contains requirements for laboratory audits and the use of performance evaluation (PE)
samples. It also contains a lab audit checklist. In addition, the QAO publication entitled "Best Practices for Detection
and Deterrence of Laboratory Fraud" includes recommendations for the selection criteria for environmental laboratories,
laboratory audits and the use of performance evaluation samples. The QAO is developing standard operating procedures
(SOPs) for the use of PE samples. If the ERO QMP is to address selection criteria for environmental laboratories,
including requirements for laboratory audits and performance evaluation samples, this QMP should make reference to
the Region-wide QMP and be consistent with existing Regional guidance and SOPs. The QAO can also provide
performance evaluation sample studies and conduct on-site laboratory evaluations upon request by the ERO.
With respect to the draft audit report's recommendation for a tracking system for monitoring laboratory
performance, there has been a directive from OERR which requires the regions to track non-CLP laboratory
performance. A direct way to address this IG recommendation and the fulfill OERR's requirement at the same time is to
provide lab performance information into the existing OERR system. The draft report also recommended that the ERO
QMP reference the OERR directive.
1 -2 Require the regional Quality Assurance Office to monitor compliance with quality management plans,
including plans for the Emergency Response Office and its technical contractor.
EPA Region 9 Response
Region 9 agrees that the QAO has responsibility for quality assurance oversight. The existing Region-wide
QMP does not explicitly define in what ways and to what extent the QAO is responsible for monitoring the compliance
36
-------
with program specific quality management plans. QAO's specific oversight roles and responsibilities in monitoring
ERO's compliance to the QMPs could be outlined in the QMP for removal actions. In addition, the QMP will identify
what organizational commitments should be in place to ensure adequate implementation for this oversight. As noted
above, the Region agrees with the recommendation that it prepare a QMP for ERO; it would be appropriate to address
these specific points in the development of that document.
The QAO in the past has provided support to ERO by performing QA oversight. One example is the QAO's
review of the ERO's QA activities as part of the 1995 Regional Quality Management System Review. The review
examined compliance with the Regional Quality Management Plan and made recommendations for improvements. The
QAO assisted the Quality Assurance Division of ORD in receiving input from several key ERO OSCs during
interviews. The resulting report incorporated prospective presentations by the OSCs.
Another example is the assessment initiated by the QAO in 1993 to ascertain whether the ERO's QA oversight
of removal actions was adequate. Copies of notes generated by the QAO staff were provided to the OIG upon request at
the inception of its audit.
Most recently, three representatives of the QAO initiated an audit of the ERO's START contractor. Started in
March, 1998, this audit meets the requirements under both the START contract as well as the Regional QMP and
addresses the need for a contract audit pointed out by the IG staff in the early stage of this audit. Additional contractor
time commitment by the ERO is necessary before the QAO can complete the audit. The QAO will make
recommendations to the ERO concerning the adherence by the contractor to the contract required quality management
and project plans.
1 -3 Amend the on-scene coordinators' responsibilities for QAPPs to include the development ofDQOs and
consideration of critical quality assurance activities. Also, ensuring responsibilities include QAPP approval,
implementation, and monitoring.
EPA Region 9 Response
Region 9 agrees with the recommendation to amend the OSCs' QA responsibilities as outlined. These
responsibilities will be included in the ERO QMP.
Although Region 9 agrees with the recommendation to amend the OSC's QA responsibilities, Region 9 also
believes that the QAO can provide considerable assistance in helping the OSCs meet those responsibilities. In those
instances in which the OSCs are not fully equipped to conduct these QA technical tasks (e.g., data validation, field
audits, performance evaluation samples, magnetic tape audits, laboratory audits, etc.), the QAO, which routinely
executes these functions, could provide significant support to the ERO.
In addition to participating in scoping sessions and meeting with ERO during the development of QAPPs and
DQOs, and reviewing and approving QAPPs, the QAO can assist ERO with difficulties encountered in the
implementation of QAPPs by providing any necessary clarifications or needed support. PE samples have proven useful
and although ERO has the lead for PE sample submission, QAO has the capacity to purchase and submit double blind
samples. Similarly, QAO has the capacity to conduct magnetic tape audits, on-site laboratory evaluations, evaluation of
data validation oversight, field audits, and data validation upon request.
1 -4 Have the Emergency Response Office and Quality Assurance Office develop a time-sensitive, team approach
for developing defensible QAPPs.
EPA Region 9 Response
Region 9 agrees with this recommendation.
QAO has designated a contact person for ERO and ERO has designated a lead contact for QAO. ERO will
involve the QAO contact in scoping for QA activities. The QAO and ERO will work to develop an agreement in the
QMP on how the two offices should interact when performing QA oversight.
37
-------
Region 9 agrees that a "time-sensitive, team approach for developing defensible QAPPs" can be used that
meets the needs of the Emergency Response Office. Although the Agency may need to accept less rigorous QA/QC
under emergency situations, timely input from the QAO can add substantial value to the quality and the defensibility of
the data used in removal actions. Substantive input can be provided in a timely manner by the QA Office during scoping
meetings. A QA specialist can work closely with a START contractor to quickly plan and document the necessary
QA/QC activities before a removal action. The QMP developed by ERO and QAO shall include procedures for timely
QAO support for those emergency responses where there is no time for planning. Even under an emergency or a time-
critical removal action, an experienced QA professional can provide on-the-spot input and existing standard operating
procedures (SOPs) for field sampling and/or analytical methods.
1 -4 Require the Quality Assurance Office to review and approve all QAPPs, including providing verbal approval
at emergency action sites, if necessary.
EPA Region 9 Response
The Emergency Response Office will work with the QAO to develop a quality management plan (QMP). The
QMP will address the roles and responsibilities of the two offices in reviewing and approving quality assurance and
sampling plans prepared by the removal contractor for emergency, time-critical, and non-time critical actions. Under the
QMP, the QAO will have the responsibility to review and approve all QAPPs. In order to accommodate the tight time
frames under which the ERO frequently operates, the QAO will work with the ERO to develop and approve generic
QAPPs for commonly recurring situations.
2-1 Require on-scene coordinators to develop DQOsfor all response actions.
EPA Region 9 Response
Region 9 agrees with this recommendation. The QMP will state the OSC's responsibility to develop DQOs for
all response actions.
2-2 Establish a minimum, mandatory training requirement for DQOsfor regional personnel, whose duties involve
the collection, evaluation, or use of environmental data.
EPA Region 9 Response
Region 9 agrees with this recommendation. The QAO, in consultation with Superfund management, will set
minimum, mandatory training requirements for OSCs and other Regional staff whose duties involve the collection,
evaluation or use of environmental data. The QAO will set these requirements by the end of the first quarter of FY99.
2-3 Require OSCs to take DQO training.
EPA Region 9 Response
Region 9 agrees with this recommendation and will seek to implement it at an early opportunity. ERO will
request that the next annual On-Scene Coordinator Readiness Training Program (RTF) sponsored by the EPA
Technology Innovation Office include this type of training. In the event such training is not available at the next annual
On-Scene Coordinator Readiness Training Program sponsored by the EPA Technology Innovation Office, Region 9 will
make alternative arrangements to assure that this training is provided. The QAO will conduct workshops for OSCs to
assure that all OSCs receive training in development of DQOs. These workshops will assure that all OSCs receive early
DQO training, in the event that DQO training is not available at the RTP or to accommodate OSCs who do not attend
the next annual RTP. At least half of the OSCs will have received DQO training by the end of the first quarter of FY99,
with all OSCs having received DQO training by the end of the second quarter of FY99.
The ERO QMP will outline the OSC's DQO training requirement. The time critical nature of an OSC's tasks
requires a special kind of DQO training different from the routine Superfund training. In general, an OSC performs
minimal exploratory sampling, but rather judgmental sampling to confirm a known occurrence. Thus, DQO training
should be tailored to provide OSCs with the skills to quickly focus on a set of data quality indicators (DQIs) from
38
-------
his/her focused removal goals. An OSC should also be trained to effectively adapt pre-developed/standard sets of DQIs
and field SOPs suitable for certain types of emergency or time critical actions. An OSC should also receive DQO
training relating to the use of rapid field sampling and analytical tools as these tools lend themselves well to removal
applications.
2-4 Use a graded or pro-forma approach to develop DQOs depending upon the scope and complexity of the
project.
EPA Region 9 Response
Region 9 agrees with this recommendation and intends to address it in the QMP.
Region 9 believes that each QAPP and each sampling/analysis plan needs to define the data quality needed and
associated QA/QC activities to ensure that adequate data quality is achieved. The needed data quality is usually based
on site-specific DQOs. However, in emergency situations, it is conceivable that a set of generic DQOs can be used for
the purpose of risk reduction. When generic DQOs are used, a rationale for using the generic rather than the site-
specific DQOs should be documented.
Furthermore, because of the nature of time-critical response actions, there may be insufficient time to develop
site-specific DQOs. Region 9's QAO has the necessary expertise and will assist in the development of generic DQOs,
FSPs, and lab and field SOPs. For non-time critical response actions, Region 9 recommends development of
standardized DQOs, QAPPs. FSPs and SOPs. Once contacted, the QAO will help develop, review, and adopt such
plans at scoping meetings.
2-5 Adopt a team approach to developing DQOs using facilitators, on-scene coordinators, quality assurance
experts, statisticians, and technical experts.
EPA Region 9 Response
Region 9 agrees that a team approach is appropriate to developing DQOs. A team approach involving OSCs
and QAO is particularly appropriate. A QAO representative would always be part of the assembled team which
develops DQOs. The team approach to DQOs will be outlined in the ERO QMP.
2-6 Designate a DQO facilitator to assist and coordinate team members through the DQO process.
EPA Region 9 Response
Region 9 agrees that the use of a facilitator could help to ensure that all perspectives and input from the team
members are considered and will identify a trained facilitator to assist in DQO development.
II. COMMENTS ON DRAFT REPORT DISCUSSION
This section contains comments on the development and use of data at removal sites. The Region believes it is
important to appreciate the distinctive role of removal actions in significant risk reduction, rather than final cleanup
which is the goal of the remedial program.
Although the Draft Report's comments are valid in identifying areas for improvement, the Region does not
believe that they invalidate the appropriateness of the OSC's judgements under the circumstances present, particularly
when one takes into account the specific role of removals in achieving significant risk reduction. Similarly, none of the
response actions taken was compromised by any of the alleged deficiencies in the DQO process.
The Region believes that it is important that differences between the remedial and removal programs should be
taken into account in assessing the DQO processes followed in the two programs. DQO guidance outlines a strategic
planning approach to preparation for data collection activities. It recognizes that the specific approach to be followed
depends in large measure on a number of parameters and that no single level of documentation or certainty is required
for all situations. It is a flexible tool that can be adapted to specific problems and situations. To cite only a few examples
39
-------
that may be appropriate for time critical removals, the Final Guidance recognizes that:
"[T]he depth and detail of DQO development will depend on the complexity of the study." Final
Guidance at p. 5.
"[T]he DQO Process is a flexible planning tool that can be used more or less intensively as the
situation requires." Final Guidance at p. 5.
• "[N]ot every problem can be evaluated using probabilistic techniques." Final Guidance at p. 7.
• The available budget needs to be taken into account. "If no design will meet the limits on decision
errors within the budget or other constraints, then the planning team will need to relax one or more
constraints." Final Guidance at p. 39.
• "Every step of this guidance may not be applicable to data collection activities where specific
decisions cannot be identified, such as studies that are exploratory in nature." Final Guidance at p.4.
The draft report states that" ...data is the basis for EPA's decision making..." In fact, particularly so for the
removal program, sampling data may be a relatively minor part of the decision matrix, and serve the purpose of filling in
details, rather than defining the basic parameters of a decision. In a given site specific situation, historical information,
visual observation and professional judgment may be much more important to decision making than the acquisition and
evaluation of sampling data. Many EPA decisions are made in a complex decision matrix, of which sampling data are
only one part.
Removal assessments are typically non-comprehensive, and are conducted with the intent of finding the points
of most concern to determine whether the action threshold should be crossed; subsequent assessment work is then
conducted as needed to fill in the details of the response. A typical drum site assessment may involve sampling only a
few drums that are flagged based on observation and judgment. If an action is subsequently taken, it is likely that the
remaining majority of the drums will receive only categorization for purposes of bulking and transport.
The most quality-critical data needs at removal actions typically arise at the end of the project, when cleanup
confirmation is desired and expected concentrations are lower, well after the most important ERO decisions have been
made. Even this may be a non-factor when we are not pursuing a complete cleanup, but instead are focussing on high
hazard/high concentration substances, with the intent of leaving residual contamination to be dealt with by another
agency or program.
40
-------
Region 9 Removal Program/ QA Office OIG Audit Recommendation
Implementation Schedule
Recommendation
1-1 Develop a QMP
1-2 Require QAO to monitor
compliance with QMP
1-3 Specify OSCQAPP
responsibilities
1-4 ERO and QAO develop a
Team approach for
QAPPs
2-1 Require OSCs to develop
DQOs
2-2 Set minimum DQO
training requirement for
all regional personnel
2-3 Require OSCs to attend
training
Region Response
Agree
Agree
Agree
Agree
Agree
Agree
Agree
Corrective Action/
Target Date
ERO and QAO will develop
ERO QMP by end of first
quarter FY99.
ERO QMP to outline QAO
oversight of Compliance by
end of first quarter FY99.
ERO QMP will specify OSC
QAPP responsibilities by end
of first quarter FY99.
ERO QMP will outline team
approach by end of first
quarter FY99; QAO to
review and approve all
QAPPs (including generic
QAPPs to help meet ERO
needs).
ERO QMP will outline
graded approach to DQO
development by end of first
quarter FY99.
ERO QMP will outline
OSC's DQO training
requirements by end of first
quarter FY 99; QAO will set
requirements for other
programs by end of first
quarter FY99.
DQO training for OSCs will
be provided by QAO by the
end of first quarter FY99.
OSCs will be required to
attend. Over half of OSCs to
be trained by end of first
quarter and all OSCs to be
trained by end of second
quarter FY99.
41
-------
2-4 Use graded approach for
DQOs.
Agree
Graded approach to DQOs
will be outlined in ERO
QMP; including time critical
as well as non-time critical
removals.
2-5 Use team approach for
DQOs
Agree
Team approach to DQOs will
be outlined in ERO QMP by
end of first quarter FY99.
2-6 Use DQO facilitator
Agree
A DQO facilitator will be
designated for ERO DQO
development by end of first
quarter FY99.
42
-------
APPENDIX B
Removal Actions Audited
Casmalia Resources
Santa Barbara
County, CA
Response Action Taken
Responsible Parties
Casmalia Resources was a hazardous waste treatment,
storage, and disposal facility which began operations in
1973. Until operations ceased in 1989, the site accepted all
types of commercial and industrial wastes including
pesticides, solvents, polychlorinated biphenyls (PCBs),
metals, acids, caustics, cyanide, and oil field wastes.
EPA inspected the site in March and July 1992 and
observed the exposure of landfill wastes and seepage of
liquids from a perimeter source control trench. This trench
was designed to collect contaminated groundwater from the
landfills.
The removal action, taken between August 1992 and
October 1996, included:
• Controlling landfill erosion by installing drainage
channels and by grading and seeding of landfill
slopes;
• Installing and operating a treatment system for
contaminated groundwater pumped from the control
trench; and,
• Disposing of concentrated contaminants pumped
from the pesticides and solvents landfill off the site.
In September 1996, a settlement was reached with some of
the responsible parties and they assumed some site
stabilization work.
Region 9's Emergency Response Office was responsible for
initiating the removal action including evaluating site
conditions, developing a site stabilization plan, and
implementing the plan.
EPA technical contractors assisted the on-scene
coordinators in the performance of the removal action. An
EPA cleanup services contractor was responsible for
43
-------
Cost of Removal Action
Cruz Ranch
Auburn, CA
Removal Action Taken
Responsible Parties
carrying out the removal actions including installing and
operating the on-site treatment system.
Sampling was performed by contractors. The technical
contractors were responsible for compiling sample results
and performing data validation.
EPA estimated the responsible parties would spend about
$30 million to stabilize the site under current agreements.
Through January 1998, EPA had collected about $2.3
million from responsible parties for cleanup costs. EPA's
cost for the action was about $17 million.
Cruz Ranch was the location of an unpermitted
electroplating operation. An EPA inspection found various
hazardous chemicals stored on the site and an area of soil
with elevated levels of chromium.
Inhaled chromium is a suspected human carcinogen.
Chronic exposure to chromium may produce adverse
effects on the liver, kidney, gastrointestinal and immune
systems.
The removal action, conducted in November 1996,
consisted of:
• Excavation and disposal of 85 cubic yards of
chromium-contaminated soil;
• Removal of drums and containers containing metal
solutions and corrosive acids and bases; and,
• Sampling of a water well and septic tank on the
property for metals, volatile organic compounds and
amenable cyanide.
Region 9's Emergency Response Office was responsible for
conducting the site assessment, developing proposed
actions, and completing the removal action.
EPA's technical contractor assisted the on-scene
coordinator during the site assessment and removal action
by developing the sampling plan, collecting samples,
performing data validation and preparing final reports.
44
-------
Cost of Removal Action
EPA's cleanup services contractor was responsible for the
removal of chemical containers, and excavation, removal
and backfill of contaminated soil.
According the modified action memorandum, the estimated
EPA cost of completing the removal action was $100,000.
Dodson Brothers Oil
Company
Montclair, CA
Removal Action Taken
Responsible Parties
The Dodson Brothers Oil Company site is located in
Montclair, San Bernardino County, California. From 1975
to 1982, the company recycled oil without a permit,
accepting spent solvents and other industrial wastes. These
wastes were then illegally treated, stored, and disposed of
on the site.
Although not listed on the Superfund National Priorities
List (NPL), the site was determined to be of NPL-caliber
based on preliminary hazard ranking system analysis.
Perchloroethylene (PCE), trichloroethylene (TCE), and
trichloroethane (TCA) were among some of the
contaminants found in elevated concentrations in the soil at
the site. These carcinogens are listed as hazardous
substances under the Resource Conservation and Recovery
Act (RCRA).
EPA's Emergency Response Office first performed an
assessment and then oversaw the removal of an
underground tank and contaminated soil. The Emergency
Response Office worked on this project from May 1994 to
February 1997.
In August 1994, EPA's technical contractor conducted an
assessment to identify the contents of an underground
storage tank and the extent of soil contamination.
From September 1995 to July 1996, the responsible parties
took the removal action. This action removed and disposed
of an underground tank and its contents, and excavated and
disposed of contaminated soil. Also, the excavations were
backfilled.
The pre-removal assessment was funded by the EPA, but
the removal action was funded by responsible parties both
voluntarily and under an EPA administrative order.
45
-------
Cost of Removal Action
EPA's cost for the removal action was about $250,000.
Region 9 did not track the funds spent by responsible
parties cleaning up the site.
Sanders Aviation
Tempe, AZ
Removal Action Taken
Responsible Parties
Tucson International
Airport
Tucson, AZ
The Sanders Aviation site is located in Tempe, Arizona.
The Sanders Aviation Company was an aerial applicator of
pesticides from 1951 to 1984. A number of pesticides were
stored, mixed, and disposed of on-site, including toxaphene
and DDT. Both of these pesticides are known animal and
probable human carcinogens, and are listed as hazardous
substances under the Resource Conservation and Recovery
Act (RCRA). DDD and DDE, derivatives of DDT, were
also detected on the site.
In 1994, the State of Arizona requested that EPA clean up
the Sanders Aviation site because of pesticide
contamination. EPA's Emergency Response Office worked
on this site from October 1994 through January 1997. The
removal action included tests of various treatment
technologies, pre-removal site assessments, and a removal
action involving excavation and treatment of about 25,000
tons of contaminated soil. The treated soil was
subsequently used as backfill.
EPA determined that the potential responsible party lacked
adequate financial resources for the cleanup. The removal
action was funded by the EPA at a cost of about $2.8
million.
In 1982, EPA added the Tucson International Airport
Superfund site to the Superfund National Priorities List.
The site has been the subject of extensive EPA
investigations since then.
As a part of a remedial investigation, polychlorinated
biphenyl (PCB) contamination was found in a residential
area adjacent to the site. PCBs are very stable chemicals
with a heavy oil-like consistency used for insulation in
transformers, capacitors, switches, and voltage regulators.
People exposed to PCBs in the air for a long time have
experienced irritation of the nose and lungs, and skin
irritations, such as acne and rashes. Some studies have
shown that babies born to women who consumed PCB-
46
-------
Removal Action Taken
Responsible Parties
Cost of Removal Action
contaminated fish had problems with their nervous systems
at birth.
In October 1996, EPA issued an unilateral administrative
order to five responsible parties for a time-critical removal
action of PCB-contaminated soil. EPA ordered the parties
to remove the PCB-contaminated soil in the residential
property and in an nearby drainage system on the airport
property. EPA used a removal action, instead of an NPL
remedial action, to hasten the cleanup.
About 10,000 tons of PCB-contaminated soil was
excavated and was transported to landfills in Arizona and
California. The excavation took place from March to May
1997.
Since the site was on Superfund's National Priorities List,
the cleanup was being managed by Region 9's Superfund
Cleanup Branch. Its Emergency Response Office was not
involved in this action.
The responsible parties hired a technical contractor to
oversee the removal of contaminated soil. The technical
contractor collected the confirmation samples during the
removal action and submitted the samples to a subcontract
laboratory for analysis.
The responsible parties paid about $685,000 for the
removal action; EPA oversight costs were about $27,000.
47
-------
(This page intentionally left blank.)
48
-------
APPENDIX C
Abbreviations
Abbreviation
ARARs
CERCLA
ODD
DDE
DDT
DQO
NCP
PCB
PE
QA
QAPP
QC
RCRA
SARA
Name
Applicable or relevant and appropriate cleanup
requirements
Comprehensive Environmental Response,
Compensation, and Liability Act
Dichlorodiphenyldichloroethane
Dichlorodiphenyldichloroethylene
Dichlorodiphenyltrichloroethane
Data quality objectives
National Contingency Plan
Polychlorinated biphenyl
Performance evaluation (samples)
Quality assurance
Quality assurance project plan
Quality control
Resource Conservation and Recovery Act
Superfund Amendments and Reauthorization Act
49
-------
(This page intentionally left blank.)
50
-------
APPENDIX D
Laws, Regulations, Directives, and Guidance
Title
CERCLA
SARA
ANSI/ASQCE4-1994
40 CFR Chapter 1, §300
EPA Order 5360
EPA Order 5360. 1
EPA QA7G-0
EPA QA/G-4
EPA QA/G-5
EPA QA/R-5
Description
Comprehensive Environmental Response, Compensation
and Liability Act of 1980
Superfund Amendments and Reauthorization Act of 1986
American National Standard, Specifications and Guidelines
for Quality Systems for Environmental Data Collection and
Environmental Technology Programs (prepared by the
American Society for Quality Control)
National Oil and Hazardous Substances Pollution
Contingency Plan
EPA Quality Manual for Environmental Programs
Policy and Program Requirements to Implement the
Mandatory Quality Assurance Program
The EPA Quality System
Guidance for Planning for Data Collection in Support of
Environmental Decision Making Using the Data Quality
Objectives Process
Guidance on Quality Assurance Project Plans
EPA Requirements for Quality Assurance Project Plans for
Environmental Data Operations
51
-------
OSWER Directives and Publications
OSWER Directive or
Guidance
9360.4-01
9360.3-02
9355.9-01
Title
Superfund Removal Procedures: Quality Assurance/Quality
Control Guidance for Removal Activities
Superfund Removal Procedures: Guidance on the Consideration
of ARARs During Removal Actions
Data Quality Objectives Process for Superfund, Interim Final
Guidance
52
-------
APPENDIX E
Definitions of Quality Assurance Activities
Data Validation
Field Audits
Laboratory Audits
Magnetic Tape
Audits
Data validation is a method for ensuring laboratory data is of
known quality. It involves reviewing data against a set of criteria
to provide assurance that data is adequate for its intended use.
EPA has data validation guidelines, known as national functional
guidelines, for its own contract lab program. According to EPA
guidelines, data validation includes a review of documentation
such as raw data, instrument printouts, chain of custody records,
and instrument calibration logs.
Field audits evaluate compliance with procedures for sample
identification, sample control, chain-of-custody, field
documentation, and sampling operations.
Laboratory audits are on-site audits designed to identify technical
areas which may cause laboratories to improperly identify or
quantitate chemicals. Audits normally evaluate a laboratory's
technical expertise, standard operating procedures, facility and
equipment sufficiency, and possible sources of sample
contamination.
On-site audits provide additional insight into a laboratory's facility,
equipment, personnel, and operating procedures. It is often
beneficial to perform an on-site audit when the laboratory is being
considered for work for which it does not have performance history
with the customer.
Audits of magnetic media are used to detect manual changes in the
electronic copy of the raw data and inconsistencies between the
electronic copy and paper copy. These audits are done in
conjunction with data audits to reconstruct an analytical run.
Electronic data, often in the form of magnetic tapes, is an output of
laboratory analyses. By obtaining magnetic tapes (or other
53
-------
Performance
Evaluation
Samples
Split Samples
electronic data) from a laboratory, audits can be conducted to help
determine:
• If the laboratory is complying with its contract;
• The integrity of the laboratory's computer systems; and,
• The appropriateness of any software editing.
Electronic tape audits are usually limited to gas chromatography
and mass spectrometry (GC/MS) data generated by certain
electronic systems. Tape audits are not currently available for
inorganic data or radio nuclides.
Performance evaluation (PE) samples are prepared by "spiking" a
known concentration of chemicals into a contaminate-free media,
such as water or soil. PE samples can be administered by two
methods: "blind" or "double-blind." When a PE sample is blind,
the laboratory is aware the sample is a PE, but does not know the
chemical concentration levels.
When a sample is double-blind, the PE sample is submitted as part
of a field sample shipment, so that the laboratory is not only
unaware of the concentration levels, it is also unaware that the
sample is a PE. A laboratory's analysis of PE samples is used to
evaluate its ability to produce accurate results.
Split samples are samples collected in the field which are divided
into two samples. One sample is sent to the contract laboratory
and the other one is sent to an independent laboratory. The results
from the two laboratories are then compared and the differences
are analyzed. Split samples can be used to verify the use of proper
analytical methodology and to detect unusual data trends.
54
-------
APPENDIX F
Hanford's "Best Practice" for Data Quality Objectives
Prograr
Project
n
s ^
Project
A
|
Project
B
Develop List of Work
1
1
Project C
Develop DQO List
1
DeclsionMakers
Review
Decision
Makers DQO
Checklist
DQO
Checklist
Interview
Issues
Scoping
Issues
Scoping
Report
1st Draft DQO
Workbook
2nd Draft DQO
Workbook
Review & Prep
Final DQO
Workbook
Planning
Documents
QAPJP; FSP
(SAP)
55
-------
This chart was prepared by Hanford Nuclear Reservation's environmental restoration contractor
(ERC), Bechtel Hanford, Inc.
Data Quality Objectives Web Site
The Department of Energy has a web site with DQO case studies of lessons learned and cost savings,
along with DQO resources. Its address is
http://etd.pnl.gov:2080/DQO/case_studies.html.
56
-------
APPENDIX G
Report Distribution
Distribution
Individual or Activity
Office of Inspector
General
Acting Inspector General (2410)
EPA Headquarters
• Assistant Administrator for Research and Development (8101)
• Assistant Administrator for Solid Waste and Emergency Response
(5101)
• Comptroller (2731)
• Associate Administrator for Regional Operations and State/Local
Relations (1501)
• Associate Administrator for Congressional and Legislative Affairs
(1301)
• Associate Administrator for Communication, Education, and Public
Affairs (1701)
• Director, National Center for Environmental Research and Quality
Assurance (8201)
• Agency Followup Official (2710)
• Agency Followup Coordinator (2724)
Region 9
• Regional Administrator
• Director, Superfund Division (SFD-1)
• Assistant Regional Administrator, Policy and Management (PMD-1)
• Chief, Emergency Response Office (SFD-6)
• Regional Quality Assurance Officer (PMD-3)
• Office of Communication and Government Relations (CGR-1)
• Audit Followup Coordinator (PMD-4)
57
-------
(This page intentionally left blank.)
58
------- |