^ Office of Inspector General
o xx-iv* 2 Audit Report
*&.
SUPERFUND
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance Program
Report No. E1SKF7-08-0011-8100240
September 30,1998
-------
Inspector General Division Central Audit Division
Conducting the Audit: Denver, Colorado
Inspector General Divisions
Contributing to the Audit: Western Audit Division
San Francisco, California
Mid-Atlantic Division
Philadelphia, Pennsylvania
Program Offices Involved: Office of Research and Development
Office of Solid Waste and Emergency
Response
Regional Offices Involved: Regions 1-10
-------
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
September 30, 1998
OFFICE OF
THE INSPECTOR GENERAL
MEMORANDUM
SUBJECT: EPA Had Not Effectively Implemented
Its Superfund Quality Assurance Program
Audit Report No. E1SKF7-08-0011-8100240
FROM: Michael Simmons I$1 Michael Simmons
Deputy Assistant Inspector General
for Internal Audits
TO: Henry Longest
Acting Assistant Administrator
for Research and Development
Timothy Fields
Acting Assistant Administrator
for Solid Waste and Emergency Response
Attached is our final report on issues that impact the Agencywide quality assurance
program and Superfund environmental data collection activities. The report includes
recommendations to the Assistant Administrators for Research and Development and for Solid
Waste and Emergency Response.
Action Required
In accordance with EPA Order 2750, the primary action official is required to provide
us with a written response to the audit report within 90 days of the final audit report date.
Since this report deals primarily with the management of the quality assurance program, we
designated the Assistant Administrator for Research and Development as the primary action
official. As such, the primary action official should take the lead in coordinating the Agency's
official response so that the 90-day timeframe is met. For corrective actions planned but not
completed by the response date, reference to specific milestone dates will assist in deciding
whether to close this report.
We have no objections to the further release of this report to the public. This audit
report contains findings that describe problems the Office of Inspector General (OIG) has
identified and corrective actions OIG recommends. This audit report represents the opinion of
OIG and the findings contained in this audit report do not necessarily represent the final EPA
position. Final determinations on matters in this audit report will be made by EPA managers
in accordance with established EPA audit resolution procedures.
In this particular audit, OIG did not measure the audited offices' performance against
-------
the standards established by the National Contingency Plan (NCP). The findings contained in
this audit report are not binding in any enforcement proceeding brought by EPA or the
Department of Justice under section 107 of the Comprehensive Environmental Response,
Compensation, and Liability Act (CERCLA) to recover costs incurred not inconsistent with
the NCP.
If your staff have any questions, please have them contact Jeff Hart, Audit Manager in
our Denver Office, at 303-312-6169, or Bill Samuel, Audit Liaison, at 202-260-3189. Please
refer to report number E1SKF7-08-0011-8100240 on any correspondence.
Attachment
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
EXECUTIVE SUMMARY
INTRODUCTION
U. S. Environmental Protection Agency (EPA) managers
are required to make important decisions on complex issues
that have significant environmental, social, health, and
economic impacts and consequences. To support these
decisions, EPA must collect data to gain a better scientific
understanding of the environmental issues being addressed.
Although the collection of the data is necessary and in many
cases required by law, it is expensive. Each year EPA and
the regulated community spend about 5 billion dollars
collecting environmental data. The EPA quality assurance
program was intended to help EPA organizations conduct
their data collection operations more efficiently and cost
effectively by incorporating scientific and systematic
planning into the process.
To date, Superfund is the only EPA program for which we
have completed quality assurance audits. Because our audit
work showed that Superfund weaknesses resulted from how
the EPA quality assurance program was implemented in the
regions, we anticipate that similar weaknesses in other EPA
program offices may exist. Accordingly, we expanded our
work to review implementation of some comprehensive
aspects of the Agency wide quality assurance program in
order to make recommendations to improve potential data
weaknesses in other programs.
OBJECTIVES
Our overall purpose was to determine if EPA had developed
and implemented a quality assurance program to ensure that
environmental data used to support decision making in the
Superfund program was of sufficient quality to satisfy the
intended purpose. Our specific objectives were:
• Has EPA effectively developed and
implemented its mandatory Agencywide
quality assurance program to obtain quality
data from Superfund field sampling?
• Is EPA appropriately implementing its policy
i Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
to develop data quality objectives to support
Superfund decision making?
Is EPA's oversight of Superfund field
sampling adequate to ensure data of known
and adequate quality?
Has EPA developed an effective quality
assurance training program?
RESULTS IN BRIEF
EPA's Quality Assurance Division and the Superfund
program had developed many critical elements necessary for
a strong and effective quality assurance program. However,
EPA managers had not demonstrated their commitment to a
cohesive, centrally-managed, mandatory Agencywide
program by fully developing and effectively implementing
the program to obtain Superfund and other data of known
and adequate quality. Because EPA needs scientifically
sound environmental data to achieve its overall mission and
effectively implement its strategic plan, EPA required a
consistent and effective quality assurance program for all
programs, including Superfund. However, the program was
not as successful as it could have been because senior EPA
managers in the Offices of Research and Development and
Solid Waste and Emergency Response and in the regions had
not always fully supported the Agencywide program by
establishing and implementing minimum project planning,
oversight, and training requirements; providing necessary
tools and resources; and asserting their authority to fully
implement the program. Without an effective Agencywide
program, EPA could not fulfill its mission which includes
ensuring environmental data of known and adequate quality.
RECOMMENDATIONS
We recommend the Acting Assistant Administrator for
Research and Development design a strategy to
institutionalize the quality assurance program; place EPA's
top quality assurance manager at an organizational level
where that individual can be an effective and independent
advocate; improve oversight, including improved
management assessments, to ensure the program is
Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
effectively implemented; develop minimum quality assurance
requirements; and report annually on Agencywide program
effectiveness.
We recommend the Acting Assistant Administrator for Solid
Waste and Emergency Response require Office of
Emergency and Remedial Response quality assurance staff to
continue performing regional management and technical
assessments to ensure that the data quality objectives policy
is being adequately implemented in the Superfund program
to improve project planning, provide Superfund staff with
sufficient tools to implement EPA's data quality objectives
policy, place the Superfund quality assurance manager at an
organizational level where that individual can be an effective
and independent advocate, and implement EPA guidance on
quality assurance training.
AGENCY COMMENTS
AND
OIG EVALUATION
With a few exceptions, EPA generally agreed with our
findings and recommendations. EPA offered comments to
clarify some issues and recommendations, and we have
modified our report as appropriate. We summarize EPA
comments at the end of each chapter highlighting those
significant issues on which we and EPA disagreed. We also
include the full text of the comments as Appendices I and II.
in
Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
TABLE OF CONTENTS
Page
EXECUTIVE SUMMARY i
ABBREVIATIONS vii
CHAPTERS
1 INTRODUCTION 1
PURPOSE 1
BACKGROUND 2
SCOPE AND METHODOLOGY 5
2 A CONSISTENTLY IMPLEMENTED QUALITY ASSURANCE PROGRAM
WOULD STRENGTHEN SUPERFUND ENVIRONMENTAL DATA 6
EPA HAS ADEQUATE AUTHORITY TO PUT IN PLACE A STRONG
PROGRAM 6
ORD HAD NOT INSTITUTIONALIZED ITS PROGRAM 7
QAD DEVELOPED MANY CRITICAL ELEMENTS OF AN
EFFECTIVE PROGRAM 10
EPA'S PROGRAM HAD NOT PROVEN FULLY EFFECTIVE 12
ORD SHOULD ASSERT ITS AUTHORITY TO PROMOTE
CONSISTENCY 17
CONCLUSION 18
RECOMMENDATIONS 18
AGENCY COMMENTS AND OIG EVALUATION 20
3 DEVELOPING PROJECT SPECIFIC DATA QUALITY OBJECTIVES WOULD
IMPROVE SUPERFUND PLANNING AND DECISION MAKING 22
iv Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
EPA REQUIRED CONFIRMED DATA QUALITY OBJECTIVES 22
MANAGERS HAD NOT CONSISTENTLY IMPLEMENTED EPA'S
DATA QUALITY OBJECTIVES POLICY 24
MANAGERS HAD NOT PROVIDED ADEQUATE DATA QUALITY
OBJECTIVES DIRECTION OR TOOLS 25
CONCLUSION 28
RECOMMENDATIONS 29
AGENCY COMMENTS AND OIGEVALUATION 30
4 INCREASED EPA OVERSIGHT WOULD STRENGTHEN SUPERFUND
QUALITY ASSURANCE EFFECTIVENESS 32
OVERSIGHT IS REQUIRED FOR AN EFFECTIVE SUPERFUND
QUALITY ASSURANCE PROGRAM 32
SENIOR SUPERFUND MANAGERS HAD NOT CONSISTENTLY
IDENTIFIED OVERSIGHT WEAKNESSES 33
SUPERFUND MANAGERS HAD NOT DEMONSTRATED QUALITY
ASSURANCE OVERSIGHT COMMITMENT 35
CONCLUSION 38
RECOMMENDATIONS 38
AGENCY COMMENTS AND OIG EVALUATION 40
5 EFFECTIVE TRAINING WOULD PROVIDE A BETTER FOUNDATION
FOR THE SUPERFUND QUALITY ASSURANCE PROGRAM 42
EPA REQUIRES QUALITY ASSURANCE TRAINING 42
OERR AND SOME REGIONAL ADMINISTRATORS HAD NOT
IDENTIFIED AND PRO VIDEO NECESSARY TRAINING 43
SOME SENIOR MANAGERS WERE NOT ACCOUNTABLE
AND DID NOT DEMONSTRATE THEIR COMMITMENT
TO SUPERFUND QUALITY ASSURANCE TRAINING 44
V Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
CONCLUSION 45
RECOMMENDATIONS 45
AGENCY COMMENTS AND OIG EVALUATION 46
EXHIBITS
1 METHODOLOGY 48
2 PRIOR AUDIT COVERAGE 50
APPENDICES
I ORD RESPONSE 54
II OSWERRESPONSE 71
III DISTRIBUTION 80
vi Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
ABBREVIATIONS
CERCLA Comprehensive Environmental Response, Compensation,
and Liability Act of 1980
CFR Code of Federal Regulations
EPA Environmental Protection Agency
OERR Office of Emergency and Remedial Response
OIG Office of Inspector General
ORD Office of Research and Development
Order EPA Order 5360.1 entitled Policy and Program Requirements
to Implement the Mandatory Quality Assurance Program,
dated April 1984
OSWER Office of Solid Waste and Emergency Response
QAD Quality Assurance Division
vn
Report No. E1SKF7-08-0011-8100240
-------
CHAPTER 1
INTRODUCTION
PURPOSE U. S. Environmental Protection Agency (EPA) managers
are required to make important decisions on complex
issues that have significant environmental, social, health,
and economic impacts and consequences. To support
these decisions, EPA must collect data to gain a better
scientific understanding of the environmental issues being
addressed. Although the collection of the data is
necessary and in many cases required by law, it is
expensive. Each year, EPA and the regulated community
spend about 5 billion dollars collecting environmental
data. The EPA quality assurance program was intended
to help EPA organizations conduct their data collection
operations more efficiently and cost effectively by
incorporating scientific and systematic planning into the
process.
Our overall purpose was to determine if EPA had
developed and implemented a quality assurance program
to ensure that environmental data used to support decision
making in the Superfund program was of sufficient quality
to satisfy the intended purpose. Our specific objectives
were:
• Has EPA effectively developed and
implemented its mandatory Agencywide
quality assurance program to obtain
quality data from Superfund field
sampling?
• Is EPA appropriately implementing its
policy to develop data quality objectives to
support Superfund decision making?
• Is EPA's oversight of Superfund field
sampling adequate to ensure data of
known and adequate quality?
• Has EPA developed an effective quality
assurance training program?
To date, Superfund is the only EPA program for which
we have completed quality assurance audits. Because our
audit work showed that Superfund weaknesses resulted
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
from how the EPA quality assurance program was
implemented in the regions, we anticipate that similar
weaknesses in other EPA program offices may exist.
Accordingly, we expanded our work to review
implementation of some comprehensive aspects of the
Agencywide program in order to make recommendations
to improve potential data weaknesses in other programs.
BACKGROUND
EPA Quality Assurance
Program
Superfund Quality
Assurance Program
In May 1979, the EPA Administrator recognized the
importance of environmental data quality by issuing a
policy statement intended to establish a centrally-
managed, mandatory Agencywide quality assurance
program. The goal of the program was to produce
environmental data that was "...scientifically valid,
defensible, and of known precision and accuracy...." This
policy was to apply to EPA and to non-EPA organizations
performing work on behalf of EPA through extramural
agreements. The Assistant Administrator for Research
and Development was delegated primary responsibility for
"...developing the national quality assurance program and
directing and coordinating its implementation...."
Program offices and regional offices were also responsible
for program implementation. The policy recognized two
primary requisites for implementation of the program.
First, the program required direct attention by top
managers for Agencywide implementation; and second,
the program needed sufficient resources and authority to
support the national program effort. The Office of
Research and Development (ORD) was to report annually
to the Administrator on the program progress and
efficacy.
In April 1984, EPA Order 5360.1 (Order), entitled Policy
and Program Requirements to Implement the Mandatory
Quality Assurance Program, was issued to guide
implementation of the policy. The stated goal of the
program was to "...ensure that all environmentally related
measurements supported by the EPA produce data of
known quality...." EPA defined data of known quality as
Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
data that was thoroughly documented, verifiable, and
defensible. The Order established ORD's Quality
Assurance Division (QAD) to serve as the "central
management authority" for this program. Specifically,
QAD is responsible for developing and ensuring the
Agencywide program is effectively implemented.
In July 1998, EPA issued a revised Order 5360.1 CHG 1
that superseded the 1984 Order. Attached to the July
1998 Order is the EPA Quality Manual that contains
mandatory requirements for implementing the new Order.
Although we based our audit on the 1984 Order, we also
have made some references to the 1998 Order.
All EPA organizations governed by the Order are
required to document their quality systems in quality
management plans. This plan is a policy statement
describing how an EPA organization will comply with the
requirements in the Order.
EPA stated in its 1997 strategic plan that it "...will
continue to implement its mandatory Quality Assurance
Program" to ensure high quality environmental data. The
plan identified one of its purposes as ensuring that,
"National efforts to reduce environmental risk are based
on the best available scientific information." The plan
recognized that in the past, significant concerns had been
expressed about the adequacy of EPA's ability to assess
risk, and that improvements were needed. The plan
indicated a need for guidance "...leading to more
defensible environmental decisions."
EPA developed a "data quality objectives" process as its
systematic planning tool to help ensure it consistently
collected the type and quality of data needed to support
technical decisions. The data quality objectives process
is based on the scientific method and thus contributes to
EPA's goal of sound science. EPA requires site-specific
project plans to include a data quality objectives section
to document the project's defined goal and planning
outputs.
The Comprehensive Environmental Response,
Compensation, and Liability Act of 1980 (CERCLA)
3 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
established the Superfund program to respond to releases
or the threat of releases of oil and hazardous substances
which threatened public health or the environment.
CERCLA was revised and expanded by the Superfund
Amendments and Reauthorization Act of 1986. Section
105 of both acts required revision and republication of the
National Contingency Plan to reflect the responsibilities
and powers created by the Superfund acts. The National
Contingency Plan established procedures and standards
for responding to the threat or actual release of oil or
hazardous substances.
The National Contingency Plan regulates the Superfund
Removal and Remedial Program through Part 300 of 40
Code of Federal Regulations (CFR). Sections 300.415,
300.420 and 300.430 require sampling and analysis plans
that provide a process for obtaining environmental data of
sufficient quality and quantity to support decision making.
EPA must review and approve sampling and analysis
plans at most Superfund sites. Sampling and analysis
plans consist of: (1) a field sampling plan, and (2) a
quality assurance project plan. A field sampling plan
describes the number, type, and location of samples and
the type of analysis required. The quality assurance
project plan describes policy, organization, and functional
activities necessary to develop adequate data. The data is
used for planning and documenting removal or remedial
activities. EPA staff often use the terms "sampling and
analysis plan," "field sampling plan," and "quality
assurance project plan" interchangeably. Throughout this
report, we have used the term "project plan" to refer to
all these types of plans.
The Superfund quality assurance program is the joint
responsibility of the Office of Solid Waste and Emergency
Response (OSWER), the Office of Emergency and
Remedial Response (OERR), and the regions. OERR is
the national program manager responsible for managing
the Superfund program in accordance with CERCLA.
OERR's October 1996 approved quality management
plan states that, "Both OERR and Regional offices are
responsible for reviewing and evaluating Superfund QA
[quality assurance] activities." OERR and regions are
responsible for tailoring principal elements of the
4 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
Agencywide program to the Superfund program.
OERR's quality management plan documents its quality
assurance program for Superfund environmental data
collection activities and outlines the program's policies,
requirements, organizational responsibilities, and
functional structure. Specifically, the plan stated that the
OERR quality assurance program goals would be
"...achieved through proper planning, organization,
review, communication of objectives, auditing, reporting,
and corrective action."
SCOPE AND We conducted seven regional audits in Regions 3, 8, 9,
METHODOLOGY and 10 that examined some aspect of Superfund field
sampling quality assurance. We also reviewed quality
management plans and management assessments for all
10 regions that described and evaluated regional quality
assurance programs. As a result of Superfund quality
assurance problems that had national implications, we
expanded our audit to examine the framework of the
quality assurance program and its implementation in
regional Superfund offices. Therefore, we conducted
work at ORD's QAD and OSWER's OERR. We also
supplemented our previous regional audit work using
results from a questionnaire sent to all regional quality
assurance managers. We conducted our work between
April 1997 and July 1998. We performed our audit in
accordance with Government Auditing Standards (1994
Revision) issued by the Comptroller General of the
United States.
See Exhibit 1 for methodology details. See Exhibit 2 for
prior audit coverage.
Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
CHAPTER 2
A CONSISTENTLY IMPLEMENTED QUALITY
ASSURANCE PROGRAM WOULD STRENGTHEN
SUPERFUND ENVIRONMENTAL DATA
EPA had not institutionalized a cohesive, centrally-
managed, mandatory Agencywide quality assurance
program that was consistently implemented. Although
QAD had developed many critical elements necessary for
an effective program, it had not fully developed and
implemented the program or reported on the program's
effectiveness. The Order makes QAD responsible for
developing requirements and overseeing implementation
of EPA's centrally-managed, mandatory program.
However, senior ORD managers had not asserted their
authority by requiring all EPA organizations to comply
with specific, minimum requirements. As a result, EPA
did not have a fully effective Agencywide program, and
EPA senior managers did not consistently ensure that data
collected in support of Superfund decisions and perhaps
other programs was of known and adequate quality.
EPA HAS ADEQUATE
AUTHORITY TO PUT
IN PLACE A STRONG
PROGRAM
ORD's Assistant Administrator has authority to establish,
direct, and coordinate a centrally-managed Agencywide
quality assurance program. EPA's 1979 policy statement
and 1984 Order clearly established the intent to develop
and implement a centrally-managed Agencywide program
that would promote national consistency in data quality.
The July 1998 Order remains consistent with the 1984
Order regarding a centrally-managed, mandatory
Agencywide program. The July 1998 Order states, "The
Quality Assurance Division (QAD) is designated by the
AA [Assistant Administrator]/ORD to serve as the central
management authority for this [quality assurance]
program."
Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
40 CFRPart 1 describes EPA's organization and provides
general quality assurance information. 40 CFR Part 1,
SubpartB, Section 1.45 authorizes ORD, as the
headquarters program office, to put in place necessary
tools for EPA organizations to implement a strong quality
assurance program. 40 CFRPart 1, Subpart A, Section
1.5 describes EPA's organization and responsibilities.
EPA's "...basic organization consists of Headquarters and
10 Regional Offices." Headquarters maintains overall
planning, coordination, and control of EPA programs,
while regions are responsible for executing the programs.
EPA has about 40 internal organizations in addition to
states, tribes, and other agencies that must comply with
EPA's quality assurance program requirements.
ORD HAD NOT
INSTITUTIONALIZED
ITS PROGRAM
EPA Lacked A Documented
Quality Assurance Strategic
Plan
ORD had not institutionalized its program by ensuring
QAD had a strategic plan that addressed EPA's sound
science program goal and objectives, and by providing
adequate staff and resources at a level of authority and
visibility to effectively develop and implement the
program.
EPA lacked a quality assurance strategic plan describing
the program's mission and goals. Such a plan would serve
as a needed framework for quality assurance planning and
resource allocation decisions. Because EPA relies on
environmental data to make decisions that impact human
health and the environment, EPA's September 1997
Strategic Plan recognized the need for a mandatory
Agencywide quality assurance program. However,
ORD's overall strategic plan did not fully identify
Agencywide program goals. Also, although QAD is
tasked with developing and implementing the Agencywide
program, it did not have its own specific strategic plan that
identified Agencywide program goals. The QAD Director
agreed that QAD would benefit from developing a
strategic plan, but competing priorities prevented
preparing one. Also, ORD did not require QAD to
develop Government Performance and Results Act goals,
objectives, and sub-objectives because Q AD's budget did
not meet ORD's dollar threshold for requiring such
performance measures.
Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
Staffing Levels Did Not
Demonstrate Program
Commitment
A QAD strategic plan would provide all quality assurance
managers and program staff the direction necessary to
implement a cohesive Agencywide program. The plan
would establish the national strategies for achieving
quality assurance goals and objectives in the future. QAD
should also identify the resources needed to accomplish
QAD's performance goals and measures used to assess
whether the goals are met. Without a plan, QAD could
not determine the adequacy of its resources.
Regional staff perceived that EPA senior managers were not
committed to the quality assurance program because senior
managers had not provided QAD adequate resources to fully
develop and assess the effectiveness of the program.
Regional staff also perceived that ORD senior managers had
not ensured the program was appropriately staffed to fulfill
its responsibilities including developing and issuing policy
and guidance documents, developing generic quality
assurance training materials, and conducting and issuing
timely management assessments.
The results of a questionnaire we sent to all regional quality
assurance managers showed that staff perceived that QAD
lacked senior management support and national leadership in
quality assurance due to what regional staff perceived was a
low level of resources invested in the program. The QAD
Director stated that staff resources have been strained as a
result of QAD being given additional tasks. As of March
1998, the proposed QAD staffing plan included 20 staff
positions of which 17 were filled to conduct work in the
areas of: (1) quality assurance, (2) peer review, and (3) the
National Environmental Laboratory Accreditation Program.
However, without a workload model or divisional strategic
plan, we were unable to assess the adequacy of QAD's
resources to accomplish its goals.
The QAD Director stated that lack of adequate resources
prevented QAD from developing and timely issuing
adequate guidance to establish an effective program. In
response to our questionnaire, four regional quality
assurance managers responded that guidance documents
had been untimely and some were still draft. For example,
although QAD's training guidance document QA/G-10,
Quality Assurance Training for Environmental Data
8
Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
Organizational Placement Did
Not Demonstrate
Program Commitment
Operations, was a critical component of the program, it was
in draft form in 1996 and still not finalized in September
1998.
QAD's training program and management assessment
program suffered when staff were reassigned from quality
assurance activities to peer review activities. In February
1997, at the request of the Deputy Administrator, QAD
staff conducted an evaluation of "the implementation of the
Agency's peer review policy." The peer review evaluation
occurred at the same time that the QAD staff were trying to
prepare for QAD's National Quality Assurance Training
that was held in August 1997. As a result, some QAD staff
were not properly prepared for their presentations at the
training conference and did not clearly communicate quality
assurance concepts, principles, and requirements.
ORD senior managers had not demonstrated their
commitment to the quality assurance program by ensuring
that QAD was organizationally located at an appropriate
level to timely and effectively accomplish its responsibilities.
The July 1998 Order states that all quality assurance
managers should function "independently" and
".. .report[s].. .to the senior manager having executive
leadership authority for the organization...." A July 1998
EPA Science Advisory Board report agreed that "quality
assurance managers are most effective when they report to
the highest career professional in an organization...".
Although QAD is responsible for developing and
implementing an Agencywide program, QAD reports to a
manager one level below the ORD Assistant Administrator.
In our opinion, QAD may not be organizationally located
within the reporting chain at the proper level to accomplish
its goals.
Regional EPA quality assurance staff had the perception that
quality assurance importance was diminished because QAD
and regional quality assurance program offices were
inappropriately located and staffed within their respective
organizations. In response to our questionnaire, some
regional quality assurance managers stated that QAD and
their regional offices were located within organizations with
little visibility or apparent authority. Regional quality
assurance managers further commented that QAD was not
9 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
perceived to have any real authority to direct a centrally-
managed program and could be ignored. Regional quality
assurance staff perceived the program lacked support for
adequate resources because it was not a media program
with an independent source of funding. For example,
QAD's management assessments reported that in 7 of 10
regions, the quality assurance managers experienced
difficulty in negotiating and securing adequate resources for
the program from the media program offices.
In our opinion, QAD was unable to ensure the revised July
1998 Order and manual were issued promptly because QAD
was not placed at an organizational level high enough to
ensure timely comments and issuance. For example, QAD
began circulating the revised Order and EPA Quality
Manual in mid-1995 for all quality assurance managers' and
other top EPA managers' review, but QAD was unable to
obtain final management comments and complete revisions
until mid-1997. QAD did not obtain final approvals for
these documents until late July 1998.
QAD DEVELOPED MANY
CRITICAL ELEMENTS
OF AN EFFECTIVE
PROGRAM
Despite competing priorities and organizational barriers,
QAD developed many critical elements necessary for a
strong and effective quality assurance program. QAD
developed and: (1) continually worked to improve its
preferred systematic planning approach, namely the data
quality objectives process; (2) issued many of its planned
requirements and guidance documents; and (3) disseminated
training modules and provided training assistance.
Since 1984 when EPA first introduced the requirement for
data quality objectives, QAD has worked diligently to
develop and improve its preferred 7-step data quality
objectives process. The process is a systematic planning
approach to developing data quality objectives for project
managers to determine the type, quantity, and quality of
data needed to support EPA decisions. QAD has continued
evolving the process at the request of EPA project
managers who wanted a more user-friendly process.
10
Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
In addition to the July 1998 Order and manual, QAD has
developed and issued other policy and guidance documents
necessary to implement a strong and effective quality
assurance program. As of April 1998, QAD developed 17
requirements and guidance documents to address program
needs. Requirements documents are designed for the
non-EPA community that is subject to EPA's quality
assurance program through appropriate extramural
regulations. QAD developed three requirements documents
but issued none in final form. Guidance documents are
supplemental documents describing suggested methods for
implementing the requirements for both EPA and non-EPA
organizations. QAD developed 14 guidance documents and
issued 6 in final form.
QAD developed and disseminated extensive quality
assurance training materials. The training modules
consisted of briefing slides but did not include speaker
notes. QAD is in the process of developing speaker notes
to further assist training presenters. QAD had also fully
developed six training courses. For its March 1998 national
training held in Denver, Colorado, QAD developed a
comprehensive project planning course that walked trainees
through the data quality objectives process. In response to
our questionnaire, regional quality assurance managers
commented that QAD had been responsive to their requests
for training assistance.
EPA corrected the fiscal 1992 material weakness regarding
environmental data quality in fiscal 1997 primarily by
ensuring adequately documented organizational quality
management plans existed in each EPA organization,
developing model generic quality assurance performance
standards for managers, and issuing a revised quality
assurance Order and manual. However, as discussed below,
QAD must take additional actions to establish a fully
effective program. In addition, as discussed in Chapters 3,
4, and 5, substantive implementation problems remain in the
regions and perhaps in other EPA organizations.
11 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
EPA'S PROGRAM HAD
NOT PROVEN FULLY
EFFECTIVE
QAD Did Not Adequately
Specify Minimum
Requirements
QAD had not adequately specified minimum EPA quality
assurance program requirements or effectively overseen
program implementation to ensure that EPA had a cohesive,
mandatory Agencywide program. In addition, QAD did not
annually report to the Administrator on the "progress and
efficacy" in implementing the Agencywide program as
required by EPA policy.
Though tasked with developing and assessing the
effectiveness of the Agencywide quality assurance program,
QAD had not adequately specified minimum program
requirements and had allowed individual EPA organizations
too much flexibility in implementing the program.
Developing a sound foundation for a strong program
includes both developing the overall program framework and
specifying certain minimum requirements. QAD referred to
its requirements as guidance. We found that some regional
Superfund program offices did not effectively implement
QAD's guidance partly because many elements were not
required. (Specific Superfund examples follow in Chapters
3, 4, and 5). The QAD Director stated that the July 1998
Order and manual should improve program effectiveness in
the future because these new documents clearly state the
program requirements. We agree that the new documents
now require much of what had been optional.
QAD had not required minimum quality assurance elements
be followed by EPA organizations. For example, although
QAD spent over 10 years improving and refining its 7-step
data quality objectives process, it did not require and had not
persuaded all EPA organizations to use this preferred
process as intended. QAD's 7-step process is flexible and
provides a systematic planning approach that would benefit
EPA organizations if used as intended. Also, QAD had not
required EPA organizations to list the minimum training
courses required for each EPA employee to ensure a basic
quality assurance understanding. Because the quality
management plan is a planning and management tool, listing
minimum training courses would provide managers a gauge
with which to measure minimum required quality assurance
skills. To their credit, some regional offices took the
initiative and listed specific training courses as required in
their regional plans.
12
Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
QAD may have unintentionally obscured its requirements
regarding data quality objectives in its July 1998 revision of
the Order. Data quality objectives, not to be confused with
the "data quality objectives process" which has always been
optional, have been mandatory since 1984. In a May 1984
memorandum with attachment, the Deputy Administrator
required under the EPA Order that data quality objectives
be a "necessary element" in data collection project plans.
However, EPA's revised July 1998 Order made no mention
of data quality objectives or of EPA's preferred data quality
objectives process. The QAD Director explained that QAD
chose to replace the term "data quality objectives" in the
Order with "acceptance or performance criteria" to avoid
confusion between mandatory objectives and the optional
objectives process.
Although QAD had authority to make the data quality
objectives process mandatory for all programs, both the
initial and July 1998 Orders were silent about data quality
objectives. Both Orders delegated all authority for
overseeing and ensuring implementation of an Agencywide
quality assurance program to the ORD Assistant
Administrator and the QAD Director. In commenting on a
draft of the new Order, we suggested in a January 12, 1998,
memorandum to the EPA Central Directives Officer,
Organization and Management Consulting Services, that the
Agency mandate use of the data quality objectives process
and specify the process requirements in the new Order.
QAD agreed that all environmental measurements and data
collection must be based on a systematic planning approach
that provides sound, clearly-defined criteria and objectives
so that the usability of the data can be determined. Although
QAD stated that data quality objectives (referred to as
"acceptance or performance criteria" in the July 1998 Order)
are needed for EPA data collection activities, the Director
explained that QAD chose not to make the objectives
process mandatory for all EPA programs because some
steps in the process did not apply to all EPA programs.
Instead, the July 1998 Order allows programs the flexibility
of using any "systematic planning approach" to develop data
quality objectives as long as the process meets certain
criteria.
13 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
QAD Did Not Effectively
Oversee Program
Implementation
Although the objectives process is mandatory in the
Superfund program, we found, as discussed in Chapter 3,
that Superfund staff generally did not follow the process or
a comparable planning process. We have serious concerns
that allowing more freedom of decision in a process that has
never been clearly understood or implemented is a serious
step backward for ensuring consistent collection of quality
data in EPA programs.
QAD had not effectively fulfilled some quality assurance
oversight responsibilities. QAD's quality assurance
responsibilities included approving EPA organizations'
quality management plans, conducting and documenting
management assessments to confirm compliance with the
plans, and tracking corrective actions. However, QAD
approved quality management plans that did not require
specific training courses or use of the data quality objectives
process or alternative systematic planning process with
defined criteria. QAD also approved a quality management
plan that showed quality assurance staff reporting directly to
a manager responsible for data collection activities (a
potential conflict of interest) and, in five cases,
geographically separated from the regional program staff.
Geographic separation would make it difficult for quality
assurance staff to timely review project plans or provide
other technical assistance. In response to our questionnaire,
regional quality assurance managers stated that they
perceived that geographic separation de-emphasized the
importance of quality assurance. Although QAD staff
performed management assessments, QAD staff had not
confirmed that organizations were using data quality
objectives, an important performance aspect of the quality
assurance program. QAD acknowledged and regional
quality assurance managers were concerned that QAD's
final management assessment reports were untimely.
Regional quality assurance managers also stated that they
perceived QAD's untimely management assessment reports
as further evidence that quality assurance was not really
important. QAD staff stated that they had not tracked
management assessment corrective actions.
QAD's organizational management assessments, conducted
during fiscal 1994-1997, identified concerns, but QAD did
not always use the information to effectively oversee the
14
Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
Agencywide implementation of the quality assurance
program. QAD reported findings in its management
assessments under two categories. QAD reported some
findings as "sufficiently significant" that required corrective
action and other findings as "minor" which did not require
corrective action. Major findings QAD identified included
inadequate training, weak state programs, limited resources,
non-compliance with quality management plans, no
systematic planning, and inadequate regional quality
assurance assessments. QAD's management assessments
found all these major weaknesses were evident in two
regions. In our opinion, the two regions did not have
adequate programs and QAD should have recommended the
regions report their programs as weaknesses under the
Federal Managers' Financial Integrity Act process. All 10
regions had at least one of these major weaknesses identified
in their programs. Regional quality assurance programs
appear even less acceptable considering collectively QAD's
sufficiently significant and minor findings.
QAD Had Not Reported On Although QAD performed management assessments of EPA
The Program's Effectiveness organizations, it had not summarized the results of its work
to report on the effectiveness of the program's Agencywide
implementation in accordance with a January 1979 policy
statement. The policy required "...annual, or more frequent,
reports to the Administrator on the program progress and
efficacy." The QAD Director explained that while QAD
had not separately reported to the Administrator on the
overall effectiveness of the program, QAD had reported
program implementation weaknesses to EPA's senior
leadership council through the Federal Managers' Financial
Integrity Act reporting process. However, except for the
fiscal 1996 report, these reports did not meet the annual
reporting requirement discussed above or include a summary
of the results of QAD's organizational management
assessments. QAD conducted individual EPA
organizational management assessments and reported its
findings to the respective organizations. Had QAD
summarized its findings, it would have reported to the
Administrator that in 9 of 10 regions, staff did not
understand basic quality assurance policy, practices, and
procedures. Although the QAD Director stated that annual
reporting to the Administrator is no longer required, we
could find no evidence that EPA had rescinded the 1979
15 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
policy. Furthermore, we believe the 1979 policy remains
appropriate and necessary.
QAD had at least two reporting means available. QAD
could have: (1) summarized and reported on the collective
results of its individual management assessments each time it
completed a review cycle, and (2) used some of the results
reported by EPA organizations in their Quality Assurance
Annual Report and Work Plan reports. The QAD Director
explained that EPA organizations' annual reports were
generally incomplete and not comparable until 1996 or
1997. However, ORD used such a collective reporting
approach before in December 1992 to report environmental
data quality as a material weakness.
ORD SHOULD ASSERT
ITS AUTHORITY TO
PROMOTE
CONSISTENCY
ORD senior managers had not asserted their authority and
supported QAD to require all EPA organizations to comply
with specific quality assurance requirements. Although 40
CFR and the EPA Order authorized ORD to establish,
direct, and coordinate the Agencywide program, it appeared
that ORD managers chose not to assert their authority and
have QAD impose requirements on EPA organizations,
particularly EPA regions. EPA has about 40 internal
organizations in addition to states, tribes, and others. Each
of the organizations was developing and implementing
potentially separate programs because QAD had not clearly
established minimum Agencywide requirements necessary to
institutionalize a cohesive quality assurance program.
Program managers could not be held accountable for
minimum performance requirements because QAD had not
established firm program requirements or clearly identified
QAD expectations for organizations' quality management
plans.
The QAD Director stated that whether QAD had a fully
developed Agencywide quality assurance program or not,
EPA's accountability systems are such that QAD did not
have the authority to make other organizations implement
the program requirements. In our opinion, the Agency
16
Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
cannot have an effective program unless ORD and QAD
assert their authority and establish clear quality assurance
requirements.
ORD acknowledged that during the period when the new
Order and manual were being developed and in draft form
(i.e., from May 1995 to July 1998), some EPA
organizations were using these new documents, some were
using documents from the 1980s, and others were using a
combination of both. This led to many variations in
program implementation, which ORD hopes have been
eliminated by the July 1998 issuance of the final new Order
and manual.
CONCLUSION
QAD has worked diligently and made considerable progress
in developing an effective Agencywide quality assurance
program. However, without fully developed quality
assurance requirements, EPA must assess potentially
separate quality assurance programs and cannot hold
managers accountable to implement a consistent, effective
program. QAD reported in its management assessments that
significant and widespread problems existed because each
EPA organization interpreted the program requirements
differently. However, QAD had not asserted its authority to
require greater consistency and improvements where
problems existed.
RECOMMENDATIONS
We recommend that the Acting Assistant Administrator for
Research and Development:
2-1. Develop a strategy to communicate and a
timetable to implement the requirements in
the July 1998 Order and EPA Quality
Assurance Manual throughout the Agency.
2-2. Work collaboratively with EPA organizations
to oversee and ensure consistent
implementation of a cohesive, mandatory
Agencywide quality assurance program.
17 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
2-3. To further institutionalize EPA's quality
assurance program, require QAD to develop
a strategic plan and annual performance plans
including appropriate outcome-based
performance measures and the level of
resources needed.
2-4. Place EPA's top quality assurance manager,
who is responsible for the Agencywide
quality assurance program, at an
organizational level where that individual
can: (1) be an effective and independent
advocate, as specified by the July 1998
Order; and (2) improve oversight, including
improved management assessments, to
ensure the program is effectively
implemented
2-5. Recommend to each regional administrator
and national program manager that they
review the most recent QAD and regional
quality assurance management assessments
for their unit conducted by QAD, national
program managers, and regional quality
assurance groups to determine whether the
regional administrator or national program
manager should identify quality assurance
implementation as a weakness in their unit as
part of the Federal Managers Financial
Integrity Act process.
2-6. Issue a memorandum to clarify that
"acceptance or performance criteria" ,as
referred to in the July 1998 Order, were
formerly known and referred to as "data
quality objectives."
2-7. Establish and implement a method of
ensuring adherence to minimum criteria for
an adequate systematic planning process.
2-8. Establish and implement minimum training
requirements or specific courses for staff in
all EPA programs implementing or
18 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
overseeing implementation of data quality
assurance activities.
2-9. Enhance oversight conducted by QAD by:
a. Not approving EPA organizations'
quality management plans which do
not include enough specifics to
ensure a sound quality program.
b. Verifying that EPA organizations are
actually using the data quality
objectives process or an adequate
alternative systematic planning
process to identify environmental
data needs.
c. Preparing and issuing timely
management assessment reports.
d. Tracking organizations'
implementation of corrective actions
identified in management assessment
reports.
2-10. To elevate the visibility of quality assurance
progress in support of the Agency's stated
goal of sound science, prepare an annual
report to the Administrator on the "progress
and efficacy" in implementing the
Agencywide quality assurance program.
AGENCY COMMENTS EPA generally agreed with our findings and
AND OIG EVALUATION recommendations. EPA offered comments to clarify some
issues and recommendations, and we have modified our
report as appropriate. We highlighted below those
significant issues on which we and EPA disagreed. We also
included the full text of the comments as Appendices I and
II.
ORD stated that while QAD centrally manages ORD's
Agencywide quality assurance responsibilities, QAD is not
19 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
responsible for centrally managing the Agency's quality
assurance program. We find this statement confusing and
contradictory. We agree ORD/QAD should not be
expected to implement the program on a day-to-day basis.
However, we strongly believe that EPA policy statements
beginning in 1979 and continuing through the July 1998
Order clearly demonstrate that ORD is responsible for
developing and implementing a single, centrally-managed
program. By "implementing," we mean that we believe
ORD and QAD clearly have the authority to establish and
should establish firm, mandatory, minimum Agencywide
requirements including but not limited to requiring certain
planning processes and specific training courses for certain
staff. ORD and QAD can and should ensure requirements
are implemented by a variety of methods including but not
limited to the tracking and reporting of non-compliance to
appropriate management officials.
ORD disagreed that QAD was not placed at the appropriate
level within ORD. ORD stated that QAD staff have
"Access to senior ORD officials...whenever there is a need
to work with senior level managers in other organizations."
We believe that the July 1998 Order clearly requires that the
senior EPA quality assurance manager (currently designated
as the Director, QAD) report directly to either the EPA
Assistant Administrator for Research and Development, or
possibly the EPA Administrator. The July 1998 EPA
Science Advisory Board reached a similar conclusion.
20 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
CHAPTER 3
DEVELOPING PROJECT SPECIFIC DATA QUALITY
OBJECTIVES WOULD IMPROVE SUPERFUND
PLANNING AND DECISION MAKING
Senior EPA managers had not consistently implemented
EPA's policy to develop project specific data quality
objectives for Superfund environmental data collection
activities. EPA policy and the Superfund National
Contingency Plan required EPA to develop data quality
objectives for data collection activities. The policy was only
minimally successful in the Superfund program because
QAD and OERR managers and regional administrators had
not provided staff sufficient direction or tools to implement
EPA's data quality objectives policy. Without the consistent
use of a systematic planning process with clear criteria, such
as EPA's 7-step data quality objectives process, EPA cannot
ensure that staff planning adequately and appropriately
defined the Superfund data type and quality needed.
EPA REQUIRED
CONFIRMED DATA
QUALITY OBJECTIVES
An EPA policy memorandum governing the Agencywide
quality assurance program required project specific data
quality objectives and management assessments to confirm
their use. In May 1984, 1- month after the Order was
finalized, an EPA policy memorandum specifically
required national program managers to develop data
quality objectives for data collection activities effective
September 1984. The policy stated, "Data quality
objectives are descriptors of the quality of data needed to
support a specific environmental decision or action." The
policy specifically required QAD to perform management
assessments to ensure that program offices had established
and were using data quality objectives. OERR also
required its quality assurance manager to conduct
management assessments to ensure compliance with the
policy.
Although EPA's July 1998 Order does not make specific
reference to the term "data quality objectives," the
21
Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
Superfund National Contingency Plan still requires
Superfund project plans to include data quality objectives.
Specifically, the National Contingency Plan, in Parts
300.415, 300.420, and 300.430, required that project
plans describe the "...policy, organization, and functional
activities and the data quality objectives and measures
necessary..." to achieve the intended purpose of each
Superfund data collection activity.
Both QAD and OERR issued guidance on a data quality
objectives process to facilitate decision making. In
September 1994, QAD finalized EPA QA/G-4, Guidance
for the Data Quality Objectives Process. The guidance
outlined a 7-step data quality objectives process
emphasizing "...a strategic planning approach based on the
Scientific Method...." OERR's Superfund data quality
objectives guidance followed the same presentation as the
QAD data quality objectives process and promoted use of
the 7-step process as described below:
DATA QUALITY OBJECTIVES PROCESS
1 State the problem.
2 Identify the decision.
3 Identify inputs to the decision.
4 Define the study boundaries.
5 Develop a decision rule.
6 Specify limits on decision errors.
Z__o^^
QAD, OERR, and regional administrators have overall
responsibility to ensure data quality objectives are included
in project plans for environmental data collection activities.
MANAGERS HAD NOT
CONSISTENTLY
IMPLEMENTED
EPA'S DATA QUALITY
OBJECTIVES POLICY
Senior EPA managers had not consistently implemented
EPA's policy to develop data quality objectives for all
Superfund environmental data collection activities. Office of
Inspector General (OIG) and other evaluations showed that
EPA's Superfund program had not fully developed and
22
Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
Evaluations Disclosed Data
Quality Objectives
Policy Was Not Fully
Implemented
QAD, OERR, And Regional
Administrators Had Not
Identified Data Quality
Objectives Were Lacking
implemented project specific data quality objectives as an
integral part of a Superfund site's planning process.
Although QAD, OERR, and regional administrators had
overall responsibility to ensure such objectives were the
foundation of environmental data collection activities, none
had ensured through management assessments that the
objectives were adequate to achieve a project's intended
purpose. The QAD Director told us QAD had never
intended to conduct technical assessments of projects'
specific data quality objectives.
Although EPA policy required project specific data quality
objectives since 1984, OIG audits and evaluations showed
that the Superfund program had not fully implemented the
policy. We evaluated 61 project plans and found that 48 (79
percent) showed the objectives had not been developed or
had not been adequately documented in the proj ect plans.
Two regions had QAD-approved quality management plans
that did not require data quality objectives as the foundation
of environmental data collection activities. Lead agency staff
at one federal facility site stated that they believed EPA staff
overseeing the site discouraged use of the data quality
obj ectives which hampered the lead agency's efforts to
develop objectives.
A September 1988 National Academy of Sciences multi-year
study of EPA's quality assurance program identified data
quality objectives concerns that we confirmed have continued
after approximately 10 years. The National Academy of
Sciences report endorsed EPA's accelerated institutionalism
of the data quality objectives process and recommended
"...holding management accountable for key portions of the
QA [quality assurance] program." The report criticized that
EPA was taking longer than was reasonable to effectively
implement data quality objectives.
Although QAD, OERR, and regional administrators had
overall responsibility to ensure data quality objectives were
the foundation of environmental data collection activities,
none ensured through management assessments that project
specific data quality objectives were adequate to achieve a
project's intended purpose. The QAD director stated that
QAD performed management assessments to confirm
regional compliance with regional quality management plans
23
Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
and not for specific performance criteria, such as data
quality objectives. The QAD director stated that evaluations
of performance were the responsibility of the program
office. OERR had competing priorities and had not invested
adequate resources to conduct management assessments
that would evaluate project specific data quality objectives.
QAD's management assessments reported that 6 of 10
regions had not performed regional management
assessments of regional program performance.
(Recommendations regarding management assessments can
be found in Chapters 2 and 4.)
MANAGERS HAD NOT
PROVIDED ADEQUATE
DATA QUALITY
OBJECTIVES
DIRECTION OR TOOLS
Managers Had Not Provided
Adequate Direction
Senior Superfund managers had not provided staff sufficient
direction or tools to develop project specific data quality
objectives. Superfund managers had not clearly
communicated the importance and necessity of data quality
objectives or the data quality objectives process as a
planning tool for environmental data collection activities, or
stressed their requirement. Superfund managers had also
not emphasized or made available the necessary tools to
promote data quality objectives, such as training and
interactive software packages.
Senior Superfund managers had not clearly communicated
the importance and necessity of data quality objectives or the
data quality objectives process as a planning tool for
environmental data collection activities. OIG audits showed
that some regional project managers did not spend the time
and resources necessary to properly plan their Superfund
projects. For example, OIG reported in Region 9 that for
the 5 removal actions reviewed, none had used the required
7-step data quality objectives process to design project
plans. Also, Region 9 analyzed about 420 samples for one
removal action that were not used to make the decision
indicated in the project plan—the suitability of leachate for
discharge. Another OIG report showed that at a California
Superfund site managed by a responsible party, EPA spent
over $2 million in oversight costs and the responsible party
spent over $100 million on studies and cleanup. However,
the project plan showed that the potentially responsible party
had not developed adequate data quality objectives during
its planning process.
24
Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
As an example of the deficiencies, we noted the data quality
objectives for one Superfund site plan clearly stated a need to
conduct a sampling study. However, the objectives did not
identify the point at which enough information would be
collected and a decision could be made. Appropriate
objectives for this project should include statements of the
problem (i.e., the project/task objectives); specific inputs
needed; a decision statement (i.e., a point at which remedial
action should be taken); risks associated with the decision, if
the critical sample results were too close to the action level;
and an optimal design.
When Superfund managers advocate quality assurance
expertise be used, project managers benefit from the
outcomes of a collaborative approach. For example, Region
4 staff stated the regional approach to developing data
quality objectives worked well for them because Region 4
required that the appropriate staff (including the quality
assurance experts) be involved up front during a project's
planning stage and that all project plans be reviewed by the
regional quality assurance experts. Because of the thorough
review process using a mixed team of appropriate staff,
Region 4 quality assurance staff stated that the Region's data
quality objectives were well documented in project plans and
adequate for the intended purpose.
Superfund managers had not stressed that data quality
objectives were required and that the EPA 7-step process
was OERR's adopted approach. Although senior
Superfund managers promoted EPA's recommended
process as the most effective way to obtain data of known
quality, some managers did not require its use or clearly
define an alternate process. OERR's data quality objectives
guidance followed the recommended process and
considered it a practical approach. The guidance also
allowed flexibility using a "graded approach" that
considered the decisions to be made and the amount of
confidence needed. Some project managers stated they
considered the EPA 7-step data quality objectives process
to be rigid, complex, burdensome, and onerous and were
unaware of the flexibility available in the process.
Managers Had Not Provided EPA managers had not emphasized or made available the
Adequate Tools necessary tools to promote data quality objectives such as
25 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
training, technical assistance including statisticians,
interactive software packages, and any other regionally-
developed tools. OIG audits showed that Superfund staff
did not have a good understanding of data quality objectives
and needed better training. In response to an OIG
questionnaire, three regional quality assurance managers
stated that their regions had not provided training. OIG
audits showed that some project managers' perceptions
were that quality assurance technical assistance was not
always readily available or constructive. Only 2 of 10
regions had statisticians available to assist Superfund project
managers in developing objectives. However, headquarters
Superfund quality assurance staff told us that statistical
expertise is available through Superfund contracts.
Although EPA developed interactive software to assist
project managers in developing data quality objectives, it had
not widely distributed the software package with
accompanying guidance. Also, at least one region developed
its own comprehensive standard operating procedures and
quality assurance manual that could assist other regions but
had not widely shared its accomplishments.
In response to data quality objectives training evaluations
received after its August 1997 training conference, QAD
changed its training format and made the training more
interactive. In August 1997, QAD presented quality
assurance training in sessions using a modular format. A
March 1998 QAD conference provided three courses:
Implementing Data Quality Objectives, Quality Assurance
Project Plans, and Data Quality Assessment. The training
was interactive and had participants develop data quality
objectives, document the objectives in the project plans, and
assess the validity of the data. The training used software
developed by QAD and its contractor and known as the
Data Quality Objectives Decision Error Feasibility Trials
software. Although the software and related guidance had
been available since September 1994, EPA had not widely
distributed the software or advocated its use to Superfund
project managers.
With senior managers' support, Region 4's quality assurance
experts developed a May 1996 Standard Operating
Procedures and Quality Assurance Manual that included
sections on data quality objectives and sampling designs and
26 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
techniques. Although Region 4 staff had access to the
regional procedures, the majority of the other regions were
unaware of Region 4's guidance.
CONCLUSION
Without consistently developing data quality objectives using
a systematic planning process with clear criteria, such as
EPA's 7-step process, EPA cannot ensure that staff planning
adequately and appropriately defined the data type and
quality needed to support Superfund decision making.
Although EPA had developed an effective data quality
objectives process, Superfund staff were poorly equipped to
use the process. Consequently, Superfund cleanup decisions
had been made which might have been approached
differently had senior Superfund managers and staff strongly
advocated the available process and tools.
RECOMMENDATIONS
We recommend:
The Acting Assistant Administrator for Research and
Development:
3-1. Develop a strategy to encourage
Agencywide use of EPA's 7-step data
quality objectives process as EPA's
recommended systematic planning process as
stated in the July 1998 EPA Quality Manual.
The Acting Assistant Administrator for Solid Waste and
Emergency Response:
3 -2. In concert with QAD, develop and implement
a plan to institutionalize the Superfund
program's data quality objectives process.
3 -3. Advocate the benefits of developing data
quality objectives using an across-
organization team approach.
27
Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
3-4. Provide Superfund staff with sufficient tools
to implement EPA's data quality objectives
policy in the Superfund program by:
a. Making data quality obj ectives
training more effective by integrating
available quality assurance courses to
provide a comprehensive
understanding of data quality
objectives.
b. Making automated tools (such as the
Data Quality Objectives Decision
Error Feasibility Trials software or
other interactive modeling software)
available to appropriate staff and
encouraging use through training and
examples of success.
c. Increasing awareness as to what
expertise is available to provide staff
technical assistance in areas such as
statistics either through realignment
of regional resources or contractor
support.
d. Sharing regionally-developed tools,
procedures, best practices, and
successes with QAD and with
headquarters and regional Superfund
staff.
3 -5. Recommend to each regional admini strator
that they review the most recent QAD,
OERR, and regional quality assurance
management assessment results for their unit
to determine whether the regional
administrator should identify quality
assurance implementation as a weakness in
their unit as part of the Federal Managers
Financial Integrity Act process.
28 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
AGENCY COMMENTS EPA generally agreed with our findings and
AND OIG EVALUATION recommendations. EPA offered comments to clarify some
issues and recommendations, and we have modified our
report as appropriate. We highlighted below those
significant issues on which we and EPA disagreed. We
also included the full text of the comments as Appendices I
and II.
OSWER stated that OERR had distributed some data
quality objectives direction and tools such as guidance and
software. However, we concluded that these actions were
insufficient to ensure program success.
OSWER stated that we had overstated the impact of the
Superfund program's lack of data quality objectives
direction and tools, and recommended that we add some
qualifiers. We disagreed and we clarified in our report that
our audit work covered all 10 regions. We believe our
audit results are representative of all 10 regions and
accurately reflect the condition of the quality assurance
program throughout the Superfund program
29 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
CHAPTER 4
INCREASED EPA OVERSIGHT WOULD
STRENGTHEN SUPERFUND QUALITY
ASSURANCE EFFECTIVENESS
Top OERR and regional managers had not uniformly
ensured staff adequately performed Superfund quality
assurance oversight to ensure environmental data was of
known and adequate quality. The Order assigns specific
oversight responsibilities to national program managers
and regional administrators. Senior Superfund managers
had not adequately or consistently demonstrated their
commitment to Superfund quality assurance oversight
activities. Without appropriate oversight of data
collection activities, EPA could not ensure data collected
at Superfund sites was of sufficient quality to support
decision making.
OVERSIGHT IS
REQUIRED FOR AN
EFFECTIVE
SUPERFUND QUALITY
ASSURANCE
PROGRAM
The Order assigns responsibilities to national program
managers and regional administrators to ensure an
appropriate level of oversight to establish an effective
Superfund quality assurance program. Specific oversight
responsibilities require that they ensure: (1) all projects
requiring environmental data collection are covered by an
acceptable, approved quality assurance project plan and
that the plan is implemented; (2) adequate management
assessments are performed to determine compliance with
quality assurance requirements; and (3) quality assurance
is an identifiable activity with resources adequate to
accomplish program goals.
The Superfund National Contingency Plan at CFR Part
300.415, 300.420, and 300.430 requires oversight
activities by requiring that EPA review and approve
sampling and analysis plans prior to conducting data
collection activities.
30
Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
SENIOR SUPERFUND
MANAGERS HAD NOT
CONSISTENTLY
IDENTIFIED
OVERSIGHT
WEAKNESSES
OERR Had Not Adequately
Fulfilled Its Quality
Assurance Oversight
Responsibilities
OERR and regional administrators had not adequately or
consistently identified quality assurance oversight
weaknesses or acted on weaknesses identified by others.
OERR had not performed adequate oversight by
conducting management assessments that would ensure
national consistency in how the Superfund quality
assurance program was implemented. OERR had also not
used the results of QAD management assessments to
confirm weaknesses were corrected and to prepare for its
own management assessments. Finally, regional
administrators had not ensured weaknesses identified by
regional staff, QAD, and OIG were corrected.
The OERR quality assurance manager served as the focal
point for developing Superfund quality assurance policies
and guidance and conducting regional Superfund quality
assurance management assessments. One objective for
conducting management assessments of regional
Superfund data collection activities was to ensure
consistency in how the regions performed quality
assurance oversight responsibilities.
OERR had not adequately fulfilled its oversight
responsibilities to conduct timely management
assessments of the 10 EPA regions' Superfund program
offices. In fiscal 1995, OERR conducted management
assessments in all 10 regional Superfund emergency
removal programs. Time-critical removals have minimal
quality assurance requirements up-front. However,
OERR had not diligently conducted management
assessments in its Superfund remedial cleanup program.
These cleanups are more long-term and non-time critical in
nature and have important quality assurance requirements.
Although OERR's quality assurance manager had
participated on two joint remedial management
assessments in 1995 with QAD, he had not conducted any
OERR-initiated remedial assessments until April 1998.
OERR's quality assurance manager prepared an internal
workplan to assess all 10 regions during fiscal 1998.
However, competing priorities caused him to delay his
assessments by several months. In April 1998, the quality
assurance manager completed the first of the planned
regional management assessments in Region 5.
31
Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
OERR Had Not Acted On
QAD Identified
Quality Assurance
Weaknesses
Completed management assessments would identify
whether Superfund staff were performing their work in
accordance with quality assurance requirements.
Although the Superfund program required the
development of data quality objectives, OIG reports
showed that 79 percent of regional project plans we
reviewed did not include adequately documented data
quality objectives or had no data quality objectives at all.
QAD management assessments also showed that regional
Superfund staff did not ensure that states, tribes, and other
external organizations had followed EPA quality
assurance requirements. EPA oversight of external
organizations is especially important because the
Superfund program relies heavily on secondary data from
these organizations. Had OERR conducted its planned
management assessments earlier, it could have identified
inconsistencies in how regions implemented the Superfund
quality assurance program and sought corrective action.
Headquarters Superfund staff told us that OSWER will
address quality assurance implementation weaknesses
through the Field and Analytical Services Teaming
Advisory Committee established in March 1998.
OERR had not used the results of past QAD management
assessments to help meet its objective to ensure
consistency in how the regions performed quality
assurance oversight responsibilities. Although QAD's
assessments addressed overall regional weaknesses,
OERR could have followed up to identify those
weaknesses that specifically affected Superfund
environmental data collection activities. QAD reported
that six regions had not performed adequate oversight. For
example, QAD heavily criticized EPA regional oversight of
state and tribal programs. Some regions had not, until
recently, required states and tribes to submit quality
management plans and had not consistently performed
oversight of state and tribal quality assurance performance.
Because OERR had not aggregated Superfund
weaknesses, it did not improve or promote a consistently
implemented Superfund quality assurance program.
32
Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
Regional Administrators
Had Not Ensured All
Previously
Identified Oversight
Weaknesses Were Corrected
Regional administrators had not ensured all weaknesses
identified by regional quality assurance managers, QAD,
and OIG were corrected. Weaknesses identified by
regional staff in their Superfund management assessments
were not corrected and continued to affect program
operations, despite having corrective action plans in place
to address the weaknesses. For example, the Region 8
quality assurance manager performed a Superfund
management assessment simultaneously with an OIG fiscal
1995 Region 8 audit of Superfund field sampling activities.
Both reports showed incomplete training and inadequate
oversight practices. OIG completed a 1998 followup
audit of these same issues and found problems continued,
despite a corrective action plan to address both the
regional quality assurance manager's and OIG's findings.
An OIG Region 9 report showed that Region 9 did not
adequately oversee a site being cleaned up by a responsible
party. The project manager did not approve the most
recent project plan and did not ensure that the potentially
responsible party complied with important conditions in
the project plan and in the guiding consent decree.
SUPERFUND
MANAGERS HAD NOT
DEMONSTRATED
QUALITY ASSURANCE
OVERSIGHT
COMMITMENT
Headquarters Superfund quality assurance staff agreed
that senior managers had not demonstrated their
commitment to quality assurance oversight by providing
necessary guidance and authority to perform an adequate
level of oversight of Superfund quality assurance
activities. OERR and regional administrators had not
ensured sufficient guidance was available and ensured
quality assurance managers were appropriately located
within their respective organizations to effect needed
changes.
OERR and Regions Lacked
National or Regional
Guidance Describing
Oversight Responsibilities
Although OERR is the national program office tasked with
developing national guidance, it had not issued specific
oversight guidance or procedures. The OERR quality
assurance manager planned to use the results of completed
management assessments to develop guidance. However,
because he had not yet completed the management
assessments, he had not determined weaknesses in
33
Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
Organizational Location Of
Quality Assurance
Managers Undermined Their
Authority
oversight activities that guidance would address. We
found that little actual guidance existed to require or guide
the project managers in their oversight role.
OIG audits showed that Superfund project managers were
unaware of regional oversight guidance that would help
them fulfill their quality assurance requirements. For
example, although Region 8 developed an oversight policy
memorandum as a corrective action to a 1995 OIG report,
the 1998 OIG followup audit showed that some project
managers still had not followed the policy and some
project managers did not recall having seen the policy.
OERR senior managers had not emphasized quality
assurance importance and provided adequate resources
and flexibility for the OERR quality assurance manager to
conduct necessary management assessments. The July
1998 Order states that each EPA organization with
environmental data or environmental technology
responsibilities must have:
A quality assurance manager (QAM), or person
assigned to an equivalent position, who functions
independently of direct environmental data
generation, model development, or technology
development responsibility; who reports on quality
issues to the senior manager having executive
leadership authority for the organization; and who
has sufficient technical and management expertise
and authority to conduct independent oversight of
and assure the implementation of the
organization's quality system in the environmental
programs of the organizations.
We interpret this statement to mean that the quality
assurance manager must be organizationally located
outside EPA operational units (e.g., Centers or Divisions)
that have responsibility for environmental data collection
and must have direct reporting authority to the senior
management in charge of overall program operation.
The OERR quality assurance manager's quality assurance
responsibilities compete with other responsibilities
34
Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
outlined in his performance agreement. He is a staff
member in 1 of 14 headquarters Superfund organizational
units (i.e., "process centers") reporting to the manager of
his unit. His responsibilities are not limited to quality
assurance as his workload includes many competing
management priorities. As the OERR quality assurance
manager, if he had unresolved concerns, the OERR quality
management plan stated that he could access the OERR
Director "...through the normal chain-of-command, but
formal direct access outside the chain-of-command is
available if needed." In our opinion, each quality
assurance manager needs open, uninhibited, direct
communication lines to their respective senior managers.
Otherwise, senior managers risk sending the message to
other Superfund staff that Superfund quality assurance is
not important. Headquarters Superfund quality assurance
staff agreed that the quality assurance function should be
completely independent of any line organization and at a
level at least equal to those managers responsible for
operations.
The OERR quality assurance manager cannot ensure
consistency in the Superfund program if the position is not
perceived by regional Superfund offices as having
adequate authority and is not at a level high enough to
effect needed changes. OERR reviewed draft proposals
about where to place various office functions prior to its
reorganization in October 1995. We reviewed one draft
proposal that suggested the OERR quality assurance
function become a separate OERR process center
reporting directly to the OERR Director. On the surface,
the draft proposal seemed to be a viable approach that
would have emphasized the significance of the Superfund
quality assurance program and goals. We did not see
other evidence to explain why this language was
eliminated. We also found that the approach may address
some Superfund authority concerns about quality
assurance importance.
Regional quality management plans showed that quality
assurance managers and staff were not appropriately
located within their respective regions to accomplish their
quality assurance objectives. In one region, the quality
assurance staff reported to a manager directly responsible
35 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
for data collection activities-a clear conflict of interest.
While in five regions, quality assurance managers were
located in the regional laboratories and not in the regional
offices, making oversight difficult. One of these
laboratories was located over 200 miles from the regional
office which may have strained communication and
undermined the quality assurance program's authority
over functions located in the regional office.
CONCLUSION
Without adequate headquarters oversight, OERR could
not ensure the Superfund quality assurance program met
its objectives for a consistently implemented program with
sufficient guidance. Without documented and appropriate
oversight of data collection activities, regions could not
ensure data collected at Superfund sites was of known
quality and sufficient for project-specific decision making.
Because the Superfund program has often been criticized
and subjected to litigation, it cannot afford to rely on data
that it had not ensured was of known and adequate quality.
EPA must perform adequate and effective oversight or
risk making inaccurate decisions that could affect human
health and the environment.
RECOMMENDATIONS
We recommend:
The Acting Assistant Administrator for Solid Waste and
Emergency Response:
4-1. Develop a plan with a realistic cycle for
OERR staff to perform management and
technical assessments in the regions.
Include in the plan the staffing resources
necessary to carry out the plan. Invite
regional Superfund and quality assurance
managers to participate as members of
OERR's assessment team.
4-2. Continue conducting joint management
assessments with QAD and obtain QAD
regional management assessment results to
36 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
ensure that regional Superfund staff
satisfactorily perform their quality
assurance activities.
4-3. Work with regional Superfund and quality
assurance staff to develop adequate
oversight procedures to ensure data is of
sufficient quality to support decision
making.
4-4. Develop an OSWER strategy to ensure
corrective actions in response to OIG
audits and EPA management assessments
are fully implemented.
4-5. Develop guidance to effectively implement
the Superfund quality assurance program
including specific quality assurance
responsibilities for regional project
managers.
4-6. Place the Superfund quality assurance
manager and staff at an organizational level
where they can be an effective and
independent advocate for the program.
4-7. Clearly and formally delegate quality
assurance program authority to the
headquarters Superfund quality assurance
manager.
The Acting Assistant Administrator for Research and
Development:
4-8. Coordinate management assessments with
the national program managers to ensure
that regional staff satisfactorily perform
their quality assurance activities. At a
minimum, provide regional management
assessment results to the appropriate
national program managers and their
respective quality assurance managers.
37 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
4-9. Address the placement of regional quality
assurance managers in management
assessments to encourage regional
administrators to place their quality
assurance managers at an organizational
level where they can be an effective and
independent advocate for the program.
AGENCY COMMENTS EPA generally agreed with our findings and
AND OIG EVALUATION recommendations. EPA offered comments to clarify some
issues and recommendations, and we have modified our
report as appropriate. We highlighted below those
significant issues on which we and EPA disagreed. We
also included the full text of the comments as Appendices I
and II.
OSWER stated that we had over-generalized in several
cases and recommended that we add some qualifiers. We
have included qualifiers where appropriate, and we
clarified in our report that our audit work covered all 10
regions. We believe our audit results are representative
of all 10 regions and accurately reflect the condition of
the quality assurance program throughout the Superfund
program.
OSWER disagreed that its Superfund quality assurance
manager was not placed at the appropriate level within
OSWER. Although OSWER stated its quality assurance
manager "has direct access to" top OERR management,
the quality assurance manager does not report directly to
the OERR Director or Deputy Director. Furthermore,
the quality assurance manager is in a unit with
environmental data responsibilities.
OSWER also stated its Analytical Operations/Data
Quality Center could continue to provide independent
technical assessments because ". . . the Analytical
Operations/Data Quality Center (AOC) was designed to
include oversight and implementation of quality assurance
activities for both CLP [Contract Laboratory Program]
and non-CLP data." However, the Analytical
Operations/Data Quality Center is responsible for
38 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
analyzing environmental data. We believe that the
organizational placement of the senior Superfund quality
assurance manager in a unit with environmental data
collection, analysis, or oversight responsibilities is a
conflict of interest. The July 1998 Order clearly intends
for the quality assurance manager position to be
independent of organizations responsible for
environmental data collection activities, as well as having
direct access to top management.
39 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
CHAPTER 5
EFFECTIVE TRAINING WOULD PROVIDE A
BETTER FOUNDATION FOR THE SUPERFUND
QUALITY ASSURANCE PROGRAM
OERR and regional administrators had not fully identified
Superfund-specific quality assurance training needs and
provided necessary training. EPA Order 5360.1 requires
national program managers and regional administrators to
identify and provide training to ensure staff directly involved
in data collection activities have an adequate understanding
of quality assurance. Senior Superfund managers were not
held accountable for fulfilling training requirements and had
not demonstrated their commitment to training. Without
assessing staff skills and fully identifying training needs, EPA
could not develop an effective quality assurance training
program to provide the skills necessary to ensure that data
collected in support of Superfund decision making was of
known and adequate quality.
EPA REQUIRES
QUALITY ASSURANCE
TRAINING
Two major EPA authorities require quality assurance
training. The Order requires national program managers and
regional administrators to identify program-specific quality
assurance training needs and to provide this training. The
Order specifically requires training "...for all levels of QA
[quality assurance] management, to assure that QA
responsibilities and requirements are understood at every
stage of project implementation." Also, the Superfund
National Contingency Plan, Section 300.120 (g) (1), requires
lead agencies to ensure that project managers are
appropriately trained to fulfill their Superfund
responsibilities. We interpreted the Superfund citation to
include quality assurance training.
Two headquarters offices oversee Superfund quality
assurance training requirements. QAD, as the quality
assurance coordinator, established requirements applicable to
all programs including Superfund. The requirements specify
that EPA organizations must describe their individual training
40
Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
requirements in their quality management plans. The plans
must"... describe the organization's process for establishing
training requirements, identifying training needs, assigning
priorities to them, and satisfying them." As the Superfund
national program manager, OERR oversees that specific
Superfund training requirements are included in regional
plans and are appropriately implemented. Headquarters
Superfund quality assurance staff agreed quality assurance
training should be mandatory.
As the accountable officials for regional quality management
plans, regional administrators must ensure that regional plans
include the necessary training requirements in accordance
with QAD's guidance and that regional Superfund staff
receives the training.
OERR AND SOME
REGIONAL
ADMINISTRATORS HAD
NOT IDENTIFIED AND
PROVIDED NECESSARY
TRAINING
OERR and some regional administrators had not fully
identified and provided the quality assurance training
necessary to ensure a consistent and effective Superfund
quality assurance program. Specifically, OERR had not
used management assessments effectively to improve the
Superfund quality assurance training program. In addition,
some regional administrators had not fully implemented an
effective Superfund quality assurance training program.
OERR had not used management assessments effectively to
identify deficiencies in the Superfund training program or to
improve training that when provided would heighten project
managers' skills. Had OERR conducted its planned regional
management assessments, it could have identified that
regional plans did not specify minimum, basic training
requirements. We found that only 4 of 10 regional plans
provided specific quality assurance training requirements for
Superfund staff. The six remaining regional plans addressed
training only generally and did not specifically identify
Superfund staff classifications that needed the training or the
type of training needed. Although OERR's quality
assurance manager developed questions designed to
determine the skills of project managers, OERR did not
provide or ensure training was provided. QAD's and
regions' internal management assessments identified similar
deficiencies. However, OERR had not collectively used the
41
Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
results of those assessments to ensure a consistently and
effectively implemented Superfund quality assurance
training program.
Some regional administrators had not fully implemented an
effective Superfund quality assurance training program to
ensure the appropriate staff developed the understanding
and skills necessary to fulfill their responsibilities and make
sound decisions. Quality management plans for 6 of 10
regions did not specify which Superfund staff classifications
were required to complete quality assurance training,
identify staff training needs, or specify minimum training
courses required. In the four regional plans that listed
specific training courses, OIG audits showed that senior
Superfund managers in at least two regions had not ensured
Superfund staff completed the required training. OIG
questionnaire results showed that regional quality assurance
training was not a high priority in all regions. For example,
three regional quality assurance managers commented that
no data quality objectives training had been provided in their
regions. Yet, a thorough understanding of data quality
objectives is key to correctly implementing the data quality
objectives process.
SOME SENIOR
MANAGERS WERE NOT
ACCOUNTABLE AND
DID NOT
DEMONSTRATE THEIR
COMMITMENT TO
SUPERFUND QUALITY
ASSURANCE TRAINING
Some senior Superfund managers were not held accountable
for fulfilling quality assurance training requirements and had
not demonstrated their commitment to an effective training
program. Although some senior Superfund managers had
not taken quality assurance training courses or ensured
Superfund staff attended necessary training, they were not
held accountable to ensure that they and staff completed
training. As a result, in Region 8, training attendance
problems that we identified in 1995 continued to be a
problem in a 1998 followup audit. Also, QAD regional
management assessments reported that in 9 of 10 regions,
staff, including Superfund, did not understand basic quality
assurance/quality control concepts, principles, and practices.
OERR and regional administrators had not demonstrated
their commitment to an effective quality assurance training
program. They also had not used available information to
work collaboratively to identify and fulfill Superfund training
needs.
42
Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
CONCLUSION
When senior managers attend management quality assurance
courses, they send a message to Superfund staff that quality
assurance training is important and a necessary tool to
enhance staff skills. However, even when regions have
criteria requiring Superfund staff to take specific training but
have not held senior Superfund managers accountable to
take or ensure staff take the required training, the incentive
for staff is diminished. Without assessing staff skills and
fully identifying training needs, OERR and regional
administrators could not develop an effective training
program to provide skills necessary to ensure that data
collected in support of Superfund decision making was of
known and adequate quality.
RECOMMENDATIONS
We recommend the Acting Assistant Administrator for Solid
Waste and Emergency Response through OERR work:
5-1. With QAD and regional administrators to
implement a Superfund quality assurance
training program which:
a. Identifies the skills necessary to
perform Superfund-specific quality
assurance responsibilities.
Includes quality assurance
performance language for appropriate
headquarters and regional staff.
Assesses each headquarters and
regional staff member's quality
assurance skills with respect to
established standards.
43
Identifies minimum Superfund quality
assurance training requirements for
headquarters and regional staff and
includes these requirements in quality
management plans.
Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
e. Ensures headquarters and regional
Superfund staff needing quality
assurance training complete required
training.
f. Assesses and modifies training based
on participants' course evaluations.
5-2. To assist regions in providing the necessary
training when regional training does not
prove entirely effective.
5-3. To provide senior OERR and regional
Superfund managers with sufficient training
for them to understand the importance and
necessity of the quality assurance program.
AGENCY COMMENTS EPA generally agreed with our findings and
AND OIG EVALUATION recommendations. EPA offered comments to clarify some
issues and recommendations, and we have modified our
report as appropriate. We highlighted below those
significant issues on which we and EPA disagreed. We
also
included the full text of the comments as Appendices I and
II.
OSWER may have misunderstood our recommendations
in this chapter in determining its corrective actions.
Therefore, we modified the recommendations to clarify
our intentions and will work with management to develop
appropriate, specific corrective actions.
ORD stated that there is no requirement for quality
management plans to include a list of specific training
courses, and we agree. However, we believe ORD and
QAD should use their authority to establish firm,
mandatory, Agencywide requirements for any activities
related to quality assurance including specific training
courses for certain staff. ORD and QAD can and should
ensure requirements are implemented by a variety of
methods including but not limited to the reporting of non-
compliance to appropriate management officials.
44 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
EXHIBIT 1
METHODOLOGY
Methodology Our first obj ective was to determine if EPA had developed
and implemented its mandatory Agencywide quality
assurance program.
Our review of the EPA quality assurance program
included:
• Review of the development of EPA
policies, procedures, and practices for the
program.
• Extensive discussions with QAD staff
including feedback on emerging audit
issues.
• Review of program guidance and
requirement documents.
• Review of QAD performance in meeting its
responsibilities to develop and oversee
implementation of the program.
• Review of OERR development and
oversight of Superfund quality assurance
programs to ensure environmental data of
known quality.
Our second objective was to determine if EPA had
implemented its policy to develop data quality objectives
to support Superfund decision making. Our review of
data quality objectives development and use included:
• Results of audits of Regions 8, 9, and 10.
• Review of national oversight performed by
QAD and OERR to ensure that national
policy to develop and use the data quality
objectives process were implemented.
• Review of all regional quality management
plans for requirements.
• Review of all QAD regional management
assessment reports.
• Assessment of responses to an OIG
questionnaire sent to all regional quality
assurance managers.
45 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
Our third objective was to determine if EPA organizations
were conducting adequate oversight of quality assurance
activities. Our review of EPA oversight of the quality
assurance program included:
• Results of audits of Regions 3, 8, 9, and 10.
• Review of national oversight performed by
QAD and OERR.
• Interviews with QAD and OERR staff
regarding oversight responsibilities.
• Review of quality management plans for
oversight procedures and practices for all
regions and OERR.
• Review of annual quality assurance reports
and work plans for oversight planned and
accomplished.
• Review of all QAD regional management
assessment reports.
Our fourth objective was to determine if EPA had
developed an effective quality assurance training program.
Our review of EPA quality assurance training included:
• Review of the training program developed
by QAD.
• Participation in and assessment of two
national training events provided by QAD.
• Interviews with QAD and OERR staff
regarding training.
• Review of quality management plans for
training requirements in each region and
OERR.
• Review of annual quality assurance reports
and work plans for training planned and
accomplished.
• Review of all QAD regional management
assessment reports.
• Assessment of responses to a questionnaire
sent to all regional quality assurance
managers regarding regional training
requirements and programs.
• Results of audits of Regions 3 and 8.
46 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
EXHIBIT 2
PRIOR AUDIT COVERAGE
Regional Audits OIG conducted seven audits since 1995 that addressed
various aspects of the EPA quality assurance program.
Three divisional OIG offices conducted audits in four
EPA regions. A brief description of the seven audits are
provided below. Complete copies of the audit reports are
available from OIG.
Environmental Data Quality at Superfund Removal
Actions in Region 9, Report No. E1SFF7-09-0058-
8100223, dated September 4, 1998, issued by the Western
Audit Division. This audit was performed to determine if
Region 9 had sufficient procedures to ensure that
Superfund environmental data was of known and
adequate quality for response actions. The audit found
that the Region had not:
• Used the complete data quality objectives process.
• Developed proj ect plans capable of preventing or
detecting inappropriate data.
• Included defensible or optimal plans for collecting
data.
• Reviewed or approved proj ect plans.
• Performed adequate oversight to ensure project
plans were implemented.
Region 8 Needs to Further Strengthen Its Superfund
Field Sampling Quality Assurance Controls, Report No.
E1SKG7-08-5001-8100082, dated March 25, 1998,
issued by the Central Audit Division. This audit was a
followup to the 1995 Region 8 report summarized above.
The audit reviewed the Region's improved procedures
and processes for review, approval, and implementation of
Superfund field sampling quality assurance activities.
Although improvements were made, the audit disclosed
that the Region:
• Continued to have unapproved proj ect plans.
• Often did not develop data quality objectives as
required by EPA national and regional policy and
47 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
by Superfund regulations.
• Had not significantly improved project managers'
oversight of field sampling.
• Had not ensured training effectiveness or that staff
attended training.
Region 3 Superfund Field Sampling Activities, Report
No. E1SKF6-03-0104-7'100140, dated March 27, 1997,
issued by the Mid-Atlantic Audit Division. This audit was
performed to determine if Region 3 had adequate quality
assurance and quality control procedures for
environmental data. The audit found that:
• The Region had taken 12 years to comply with the
requirement to prepare a quality management plan.
• Regional staff had not documented review of
sampling documents.
• Superfund staff had not completed required
training.
Laboratory Data Quality at Federal Facility Superfund
Sites, Report No. E1SKB6-09-0041-7100132, dated
March 20, 1997, issued by the Western Audit Division.
The purpose of the audit was to determine if EPA had
sufficient procedures in place to ensure that laboratory
data was of known quality under federal facility
agreements. The audit covered Regions 8, 9, and 10 and
disclosed that:
• Quality assurance proj ect plans were not well
designed to prevent and detect inappropriate data.
• Oversight of laboratory data quality needed to be
increased.
• EPA had not assessed other federal agencies'
quality systems for environmental data.
• There was no federal system to share laboratory
evaluations between agencies.
Special Review of EPA Region 9 Data Quality Oversight
at The Aerojet Superfund Site, Report No. E1SKG5-09-
0110-6400044, dated March 28, 1996, issued by the
Western Audit Division. This review was conducted to
review quality assurance procedures at a responsible
party-lead Superfund site. The audit found that:
48 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
• Data quality objectives had not been developed for
the site.
• The quality assurance project plan was not
approved by EPA as required.
• Responsible party had not complied with
important provisions of the quality assurance
project plan and with its consent decree.
• EPA had not performed oversight of the
potentially responsible party's performance.
Environmental Data Quality atDOD Superfund Sites in
Region 9, Report No. E1SKF5-09-0031-5100505, dated
September 26, 1995, issued by the Western Audit
Division. This audit was performed to determine if
Region 9 was ensuring that laboratory data was of known
and acceptable quality under Federal facility agreements
with Department of Defense. The audit found that:
Quality assurance project plans were not generally
designed to detect the laboratory quality problems.
Quality assurance proj ect plans were not always
officially approved or fully implemented.
• Region 9 did not sufficiently monitor compliance
with quality assurance project plans or perform
sufficient oversight to determine the quality of
laboratory data.
Quality assurance proj ect plan inadequacies were
not corrected after data problems were found.
Implementing Controls Would Improve Region 8's
Superfund Field Sampling A ctivities, Report No.
E1SKG4-08-0045-5400034, dated January 27, 1995,
issued by the Central Audit Division. This review was
performed to determine if Region 8's procedures and
processes for review, approval, and implementation of
Superfund field sampling activities were adequate. The
review disclosed that:
• Superfund field sampling was not effectively
performed.
• Proj ect plans were not consistently approved.
• Required training was not completed.
• EPA staff did not always perform effective
oversight.
• Timely technical assistance was not always
available.
49 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
APPENDIX I
ORD RESPONSE
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
OFFICE OF
RESEARCH AND DEVELOPMENT
September 4,1998
MEMORANDUM
SUBJECT: Response to OIG Draft Report of Audit No. E1SKF7-08-001-XXXXX
EPA Had Not Effectively Implemented Its Superfund Quality Assurance
Program
FROM: Henry L Longest II Deborah Y. Dietrich for
Acting Assi stant Admini strator (8101R)
TO: Michael Simmons
Deputy Assistant Inspector General
for Internal Audits
Office of Inspector General (2421)
Purpose
This memorandum responds to the Office of Inspector General's (OIG's) Draft Report
of Audit No. El SKF7-08-001 -XXXXX, EPA Had Not Effectively Implemented Its Superfund
Quality Assurance Program, dated August 6, 1998.
Discussion
In general, we find the draft report reasonable and we concur with most of the
recommendations for which we have cognizance. However, there are a few points which we feel
still merit clarification in the report.
First, the Office of Research and Development (ORD) is responsible for developing
quality assurance (QA) and quality control (QC) requirements and for overseeing
50 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
implementation of the Agencywide Quality System. We have assigned the operational
responsibility for these activities to the Quality Assurance Division (QAD). That is, they
centrally manage ORD's Agencywide quality-related responsibilities. They are not responsible
for centrally managing the Agency's QA program. The statements referring to QAD as the
central management authority for QA should be corrected to reflect this definition of their
responsibilities.
Second, a number of the recommendations imply that ORD has either the responsibility
for implementing the quality system across the Agency or that ORD has the authority or means
to ensure that organizations implement effective quality programs (see recommendations 2-2,
2-7,
2-8). While ORD can develop processes for organizations to use in systematic planning of
environmental monitoring programs and in developing minimum training requirements and
providing training courses to staff, ORD must rely on the senior managers in those
organizations to ensure that the tools provided are effectively utilized within their
organizations. We will work closely with senior managers across the Agency to ensure that
they understand their responsibilities and work to fully implement their quality systems.
Third, the report recommended (recommendation 2-4) that we place QAD at an
organizational level where it can be an effective and independent advocate for the QA program.
Although we considered this action, we feel that a reorganization is unwarranted. Access to
senior ORD officials is available to QAD staff whenever there is a need to work with senior
level managers in other organizations. There are a number of other actions that we will take
to improve the QA practices, and we propose to address these actions in our corrective action
plan.
We also have a number of comments which, if addressed, will improve the quality of
the report. These detailed comments are attached and speak to the primary report findings
and recommendations.
We appreciate the opportunity to respond to this draft report. We believe that your
attention to the issues identified in the report will help to improve the Agency's QA program.
Attached is ORD's corrective action plan. Should your staff have any questions, or require
additional information, please contact Nancy Wentworth on (202) 564-6830.
Attachments:
1. ORD Comments
2. Corrective Action Plan
cc: Tim Fields (5101)
Bill Samuel (2421)
Deborah Dietrich (8101R)
LekKadeli(8102R)
51 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
Attachment 1
ORD Comments on OIG Draft Report
Audit No. E1SKF7-08-001-XXXXX
EPA Had Not Effectively Implemented Its Superfund Quality Assurance Program
1. On page ii, paragraph 2, the draft report states:
We recommend the Acting Assistant Administrator for Research and Development . .
. place the EPA quality assurance manager at an organizational level where that
individual can be an effective and independent advocate; ....
RESPONSE: We request the following change to this portion of the draft report:
... place the Quality Assurance Division at an organizational level where ...
Discussion: To be consistent with the report text, we believe that the statements about the
quality assurance manager refer to the Quality Assurance Division, not an individual. See
additional comments on this recommendation and in the corrective action plan under
recommendation 2-4.
2. On page 2, paragraph 2, the draft report states:
...The Order established ORD's Quality Assurance Division (QAD) to serve as the
"central management authority" for this program. Specifically, QAD is responsible for
developing and ensuring the Agencywide program is effectively implemented.
RESPONSE: We request the following change to this portion of the draft report:
Specifically, QAD plays a central role in developing the Agencywide program and
overseeing its implementation. Senior managers are responsible for ensuring the
implementation of the program within their organizations.
Discussion: The statement about "central management" of the QA program is not intended to
require QAD to centrally manage the QA program, but that it is the single organization within
ORD that tracks and reports on the status of the Agencywide program. This change should be
made throughout the report. See also page 5, ]f 1, line 5, ]f 2, lines 3 and 7.
52 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
3. On page 6, paragraph 3, the draft report states:
A QAD strategic plan would provide quality assurance managers and program staff the
direction necessary to implement a cohesive Agencywide program. The plan would
establish the national strategies for achieving quality assurance goals and objectives in
the future. QAD should . . .
RESPONSE: We request the following change to the draft report:
A quality assurance strategic plan would provide quality assurance managers and
program staff the direction necessary to implement a cohesive Agencywide
program. The plan would establish the national priorities for achieving QA goals
and objectives in the future. QAD should ...
Discussion: The paragraph should clearly state whether the recommended strategic plan is a
QAD plan or a plan for the Agency's QA program. The plan will state the national priorities
for quality system implementation. Also, it should be noted that it is the responsibility of
organizations across the agency to actually implement the required actions.
4. On page 8, paragraph 5, the draft report states:
... Requirements documents are external policy documents describing what actions
must be taken to meet the Order's requirements by non-EPA organizations conducting
EPA work. QAD has developed three requirements documents and issued none in final
form. Guidance documents are internal policy documents describing methods for
implementing the requirements for both EPA and non-EPA organizations. . . .
RESPONSE: We request the following changes to this portion of the draft report:
Requirements documents are designed for the non-EPA community that is
subject to EPA's QA program through appropriate extramural regulations. The
requirements documents provide additional descriptive information on how to
comply with the regulations. QAD has developed three requirements documents
and issued none in final form; these documents could not have been completed
until the revised Order and Quality Manual were issued and conforming changes
made to the regulations. Guidance documents are policy documents containing
non-mandatory information that could be useful to both EPA and non-EPA
organizations....
Discussion: The Order and Quality Manual apply to EPA organizations, while regulations (40
CFR 30, 40 CFR 31, 48 CFR 15) invoke the QA requirements for non-EPA organizations.
The regulations are further clarified by the requirements documents. The guidance documents
provide additional information to whomever chooses to use them.
5. On page 9, paragraph 2, the draft report states:
53
Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
EPA corrected the fiscal 1992 Agency level material weakness regarding environmental
data quality in fiscal 1997 . . .
RESPONSE: We request the following change to this portion of the draft report:
EPA corrected the fiscal 1992 material weakness regarding environmental data
quality in fiscal 1997.... As a result, Agency organizations with incomplete QA
programs were required to declare weaknesses for those programs and report on
them individually. However, as discussed below
Discussion: FMFIA weaknesses are either material (reported to The President and Congress),
Agency-level (reported to the Administrator), or [in our case] ORD-Level (reported to the
AA/ORD). The words "Agency level material weakness" inaccurately combines two terms.
6. On page 9, paragraph 4, the draft report states:
Though tasked with developing and assessing the effectiveness of the Agencywide
quality assurance program, QAD did not adequately specify minimum program
requirements and allowed individual EPA organizations too much flexibility in
implementing the program. Developing a sound foundation for a strong program
includes both developing the overall program framework and specifying certain
minimum requirements. QAD referred to its requirements as guidance. We found that
some regional Superfund program offices did not effectively implement QAD's
guidance partly because many elements were not required. (Specific Superfund
examples follow in Chapters 3, 4, and 5). The QAD Director stated that the July 1998
Order and manual should improve program effectiveness in the future because these
new documents now require much of what had been optional.
RESPONSE: We request the following changes to this paragraph:
Though tasked with developing and assessing the effectiveness of the Agencywide
QA program, QAD has only recently specified uniform program requirements in
the new Order and Quality Manual. These new documents should eliminate
much of the variability between Agency programs that has occurred because of
the "draft" status of the documents. During the transition period while the new
documents were under preparation and review, some Regional Superfund
program offices did not effectively implement QAD's documents, partly because
many elements were not required. (Specific Superfund examples follow in
Chapters 3, 4, and 5.) The QAD Director stated that the July 1998 Order and
Manual should improve program effectiveness in the future because these new
documents clearly state the program requirements.
Discussion: While the Order and Manual were under development and being reviewed for
Agency implementation, QAD had no basis for requiring activities that were not explicitly
covered in the 1984 Order. Therefore, there has been a period of transition where some
organizations implemented the program as it was being described in the new program
54 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
documents and others adhered strictly to the 1984 requirements. Approval of the new Order
and Manual will require all organizations to adhere to the same requirements, albeit after
additional transition time.
7. On page 9, last paragraph, the draft report states:
QAD had not required minimum quality assurance elements be followed by EPA
organizations. For example, although QAD spent over 10 years improving and refining
its 7-step data quality objectives process, it did not require and had not persuaded all
EPA organizations to use this preferred process. QAD's 7-step process is flexible and
provides a systematic planning approach that would benefit EPA organizations if used
as intended. Also, QAD had not required EPA organizations to list in their quality
management plans the minimum training courses required for each EPA employee to
ensure a basic quality assurance understanding. Because the quality management plan is
a planning and management tool, listing minimum training courses would provide
managers a gauge with which to measure minimum required quality assurance skills. To
their credit, some regional offices took the initiative and listed specific training courses
as required in their regional plans.
RESPONSE: We request the following changes to this portion of the draft report:
Until the issuance of the Order and Quality Manual, QAD had not required
minimum QA elements be followed by EPA organizations. For example, ... as
intended. The revised Order and Quality Manual require organizations to state
the process that they will use to define QA-related training needs of their staff.
QAD does not require EPA organizations to list in their Quality Management
Plans the minimum training courses required for each EPA employee to ensure a
basic QA understanding. The Quality Management Plan is a planning and
management tool and listing minimum training courses could provide managers
a gauge with which to measure minimum required QA skills. To their credit,
some regional offices took the initiative and listed specific training courses as
required in their regional plans.
Discussion: As noted, the Quality Management Plan is a planning and management tool used
to define the processes on how activities would be conducted within the organization; it is not
intended to describe the details of each of the processes and procedures.
8. On page 10, paragraph 3, the draft reports states:
Although QAD had authority . . . Both Orders delegated all authority for
implementing an Agency-wide quality assurance program to the ORD Assistant
Administrator and the QAD Director . . .
RESPONSE: We request the following changes to this portion of the draft report:
Although QAD had authority ... Both Orders delegated all authority for
55 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
overseeing implementation of the Agency-wide QA program to the ORD
Assistant Administrator and the QAD Director.
Discussion: ORD and QAD have responsibility for overseeing implementation, not directly
implementing the Agency-wide QA program. The AA/ORD is, of course, responsible for
ensuring implementation of the QA program within ORD.
9. On page 10, last paragraph, the draft report states:
QAD had not effectively . . . QAD's quality assurance responsibilities included
approving EPA organizations' quality management plans, conducting and documenting
management assessments to confirm compliance with the plans, and tracking corrective
actions. However, QAD approved quality management plans that did not require
specific training courses or use of the data quality objectives process or alternative
systematic planning process with defined criteria . . .
RESPONSE: We request the following changes to this portion of the draft report:
QAD had not effectively ... QAD's quality assurance responsibilities included
approving EPA organizations' Quality Management Plans, conducting and
documenting management assessments to confirm compliance with the plans, and
tracking corrective actions. However, QAD approved Quality Management Plans
that did not require use of the data quality objectives process or alternative
systematic planning process with defined criteria ...
Discussion: The Quality Management Plan is a process and planning document, and there is no
requirement that it include a list of required courses for individuals with QA responsibilities. If
you believe it necessary, additional rationale should be provided in the report text.
10. On page 11, paragraph 2, the draft report states:
... In our opinion, the two regions did not have adequate programs and QAD should
have recommended the regions report their programs as regional material weaknesses ..
RESPONSE: We request the following change to this portion of the draft report:
... In our opinion, the two regions did not have adequate programs and QAD
should have recommended the regions report their programs as weaknesses under
the Federal Managers' Financial Integrity Act process ...
Discussion: See explanation in # 5, above. The words "regional material weakness"
inaccurately combines two types of weaknesses.
11. On page 12, paragraph 2, the draft report states:
56 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
...However, ORD used such a collective reporting approach before in November 1991
to report environmental data quality as an ORD material weakness.
RESPONSE: We request the following change to this portion of the draft report:
...However, ORD used such a collective reporting approach before in November
1991 to report environmental data quality as a material weakness.
Discussion: See explanation in #5, above. The words "ORD material weakness" inaccurately
combines two types of weaknesses.
12. On page 12, paragraph 3, the draft report states:
ORD senior managers had not asserted their authority and supported QAD to require
all EPA organizations to comply with specific quality assurance requirements.
Although 40 CFR and the EPA Order authorized ORD to establish, direct, and
coordinate the Agencywide program, it appeared that ORD managers chose not to
assert their authority and have QAD impose requirements on EPA organizations,
particularly EPA regions. EPA has about 40 internal organizations in addition to
states, tribes, and others. Each of the organizations was developing and implementing
potentially separate programs because QAD had not clearly established minimum
Agencywide requirements necessary to institutionalize a cohesive quality assurance
program. Program managers could not be held accountable for minimum performance
requirements because QAD had not established firm program requirements or clearly
identified QAD expectations for organizations' quality management plans.
RESPONSE: We suggest the following changes to this paragraph:
While QAD was developing the new Order and Quality Manual, ORD senior
managers did not assert their authority and support QAD to require all EPA
organizations to comply with specific QA requirements. Although 40 CFR and
the EPA Order authorized ORD to establish, direct, and coordinate the
Agencywide program, it appeared that, during this transition period, ORD
managers chose not to assert their authority and have QAD impose requirements
on EPA organizations, particularly EPA regions. EPA has about 40 internal
organizations in addition to states, tribes, and others. Each of the organizations
was developing and implementing potentially separate programs because at that
time, QAD had not clearly established minimum Agencywide requirements
necessary to institutionalize a QA program. Program managers could not be
held accountable for minimum performance requirements because QAD had not
established firm program requirements or clearly identified QAD expectations
for organizations' Quality Management Plans. With the issuance of the revised
Order and Quality Manual, clearer specifications for acceptable programs have
been defined.
Discussion: During the period when the Order and Manual were being developed, some
57 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
organizations were using the new material, some were using the 1980's material, and others
used a combination of both. This lead to many variations in program implementation, which
we hope, have been eliminated by issuance of the final Order and Manual.
13. On page 13, recommendation 2-2, the draft report states:
2-2. Work collaboratively with EPA organizations to oversee and ensure that a
cohesive, mandatory Agencywide quality assurance program is consistently
implemented.
RESPONSE: We suggest the following change to this recommendation:
2-2. Work collaboratively with EPA organizations to oversee the
implementation of the mandatory Agencywide QA program.
Discussion: Agency senior managers are responsible for the implementation of their QA
programs. The programs are required to contain certain components and processes, but the
details of design and implementation of the components and processes are the responsibility of
the managers and the organization. There will continue to be variability between programs
based on statutory and regulatory differences.
14. On page 13, recommendation 2-4, the draft report states:
2-4. Place QAD, which is responsible for EPA's quality assurance program, at an
organizational level where it can be an effective and independent advocate for
the program as specified by the July 1998 Order.
RESPONSE: We suggest that the recommendation be deleted.
Discussion: As noted in the attached table, ORD believes that QAD has sufficient access to
ORD senior management to function effectively.
15. On page 13, recommendation 2-5, and on page 20, paragraph 3-5, the draft report states:
2-5 . . . should identify quality assurance implementation as a material weakness in their
unit as part of the fiscal 1998 Federal Managers' Financial Integrity Act process.
RESPONSE: We request the following change to this recommendation:
2-5 .. .should identify QA implementation as a weakness as part of the fiscal 1998
Federal Managers' Financial Integrity Act process.
Discussion: See explanation in #5, above. The words "material weakness in their unit" are
confusing. Material weaknesses are reported to The President and Congress.
16. On page 14, recommendation 2-7, the draft report states:
58 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
2-7. Establish and implement a method of ensuring adherence to minimum criteria for
an adequate systematic planning process.
RESPONSE: We request that the recommendation be deleted.
Discussion: ORD has established criteria for systematic planning processes and QAD will
evaluate performance against them, but it remains the responsibility of senior managers across
the Agency to implement the planning process. This activity will be addressed by the corrective
action planned for recommendation 2-9b.
17. On page 14, recommendation 2-8, the draft report states:
2-8. Establish and implement minimum training requirements for staff in all EPA
programs implementing or overseeing implementation of data quality assurance
activities.
RESPONSE: We request the following change to this recommendation:
2-8. Establish a process for defining minimum training requirements for staff
in all EPA programs implementing or overseeing implementation of data
QA activities.
Discussion: QAD can establish a process for defining minimum training requirements for staff
with QA responsibilities, but it remains the responsibility of senior managers across the Agency
to develop the organization-specific training requirements and ensure that they are followed.
18. On page 18, paragraph 5, the draft report states:
EPA managers had not emphasized .. . OIG audits showed that project managers'
perceptions were that. . .
RESPONSE: We request the following change to the draft report:
EPA managers had not emphasized... OIG audits showed that some project
managers' perceptions were that...
Discussion: The statement is overly general and should be made more temperate.
19. On page 26, recommendation 4-8 states:
4-8. Coordinate management assessments ... At a minimum, provide regional
management assessment results to the national program managers' quality
assurance managers.
RESPONSE: We request the following change to this recommendation:
59 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
4-8. Coordinate management assessments ... At a minimum, provide regional
management assessment results to the appropriate national program
managers.
Discussion: For this recommendation to be effective, the findings and recommendations from
the regional assessments should be provided directly to the national program managers. See
also a corrective action plan for this recommendation.
20. On page 27, paragraph 3, the draft report states:
Two headquarters offices oversee Superfund quality assurance training requirements.
QAD, as the quality assurance coordinator, established requirements applicable to all
programs including Superfund. The requirements specify that EPA organizations must
describe their individual training requirements in their quality management plans. The
plans must"... describe the organization's process for establishing training
requirements, identifying training needs, assigning priorities to them, and satisfying
them." As the Superfund national program manager, OERR oversees that specific
Superfund training requirements are included in regional plans and are appropriately
implemented. Headquarters Superfund quality assurance staff agreed quality assurance
training should be mandatory.
RESPONSE: We request the following changes to the draft report:
Two headquarters offices oversee Superfund QA training requirements. QAD, as
the QA coordinator, established requirements applicable to all programs
including Superfund. The requirements specify that EPA organizations must
describe their processes for defining individual training requirements in their
Quality Management Plans. The plans must "...describe the processes and the
management and/or staff responsible for...identifying, designing, performing, and
documenting technical, quality, and project management training." (See section
3.3.4, Quality Manual.) As the Superfund national program manager, OERR
oversees that specific Superfund training requirements are included in regional
plans and are appropriately implemented. Headquarters Superfund QA staff
agreed quality assurance training should be mandatory.
Discussion: The Quality Management Plan describes the process used by an organization to
determine training needs. It is not expected to contain the specific training requirements for all
staff and managers.
60 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
21. On page 29, recommendation 5-le, the draft report states:
e. Develops and delivers an effective quality assurance training program that
meets minimum mandatory Agencywide requirements and incorporates these
requirements into minimum Superfund training requirements.
RESPONSE: We request the following change to this recommendation:
e. Develops and delivers an effective QA training program that meets the
organization's needs and incorporates these materials into Superfund's
training program.
Discussion: The Agency quality system does not establish mandatory training requirements.
Instead, it requires managers to ensure that training needs are defined within their
organizations and that those needs are met.
General comments that apply throughout document:
1. Use of the term "quality assurance manager" is sometimes confusing regarding which QA
manager is being considered. In most cases, the term is modified by "regional" or "OERR,"
but there are times when its meaning is confusing.
2. The term "regional Superfund staff is open to interpretation as to whether is includes
regional QA staffer not. Use of the term should be clarified. When comments and
recommendations are directed to regional staff, the recommendations need to clearly state that
staff with QA responsibilities are included.
3. A more common use is Agency-wide, not Agencywide.
61 Report No. E1SKF7-08-0011-8100240
-------
Attachment 2
OKD Corrective Action Plan to OIG Draft Report of Audit
No. E1SKF7-08-001-XXXXX
EPA Had Not Effectively Implemented Its Superfund Quality Assurance Program
Rec
#
Draft Report
Recommendation
Action
Official
Corrective
Action
Due
Date
2-1 Develop a strategy to communicate the
requirements in the July 1998 Order and EPA
Quality Assurance Manual throughout the
Agency.
Director, Quality
Assurance Division
(QAD)
A communication strategy detailing the background
of the QA program, the recent changes and how
they affect different user communities, and a plan
for communicating the requirements to the users
will be prepared and implemented.
12/98
2-2 Work collaboratively with EPA organizations
to oversee and ensure that a cohesive,
mandatory Agencywide quality assurance
program is consistently implemented.
AA/ORD with support
from QAD
QAD will continue to conduct management
assessments of QA program implementation and
raise successes and deficiencies to appropriate
senior managers' attention.
ongoing
2-3 To further institutionalize EPA's quality
assurance program, require QAD to develop a
strategic plan and annual performance plan
including appropriate outcome-based
performance measures and the level of
resources needed.
Director, QAD
QAD will develop a strategic plan for its areas of
responsibility, including performance measures and
resource estimates. QAD will then prepare a
performance plan for the upcoming year based on
the priorities and resources provided.
12/98
2-4 Place QAD, which is responsible for EPA's
quality assurance program, at an
organizational level where it can be an
effective and independent advocate for the
program as specified by the July 1998 Order.
AA/OJAD
We have considered this recommendation but do
not believe this action to be warranted. QAD has
access to OJAD senior managers whenever needed
for purposes of discussions with other senior
managers across the Agency.
N/A
-------
Rec
#
Draft Report
Recommendation
Action
Official
Corrective
Action
Due
Date
2-5 Recommend to each regional administrator
and national program manager that they
review the most recent QAD and regional
quality assurance management assessments for
their unit conducted by QAD, national
program managers, and regional quality
assurance groups to determine whether the
regional administrator or national program
manager should identify quality assurance
implementation as a material weakness in their
unit as part of the fiscal 1998 Federal
Managers Financial Integrity Act process.
AA/ORD
A memorandum will be sent to Assistant and
Regional Administrators directing them to review
any available management assessments (conducted
by QAD, conducted internally, conducted by OIG of
similar organizations or programs) to determine
whether they should identify any component of their
QA program as a weakness under the Integrity Act.
9/98
2-6 Issue a memorandum to clarify that
"acceptance or performance criteria" as
referred to in the July 1998 Order were
formally known and referred to as "data
quality objectives," and clearly specify
minimum Agencywide quality assurance
training and other program requirements.
AA/ORD
General agreement with recommendation.
Memoranda will be prepared and distributed:
1. Clarify equivalence of "acceptance or
performance criteria" to "data quality objectives."
2. Clarify management responsibility to define,
within their organization, minimum training
requirements for staff with QA responsibilities.
3. Clarify management responsibility for
implementing quality systems that fully meet the
requirements established in the Order and Manual.
12/98
3/99
12/98
2-7 Establish and implement a method of ensuring
adherence to minimum criteria for an adequate
systematic planning process.
AA/ORD
Recommend deletion. See actions in 2-9b. QAD
will include in its management assessments explicit
consideration of the use of systematic planning, and
will report to managers on the results of the
assessments. It is these managers who are
responsible for ensuring implementation.
Ongoing
-------
Rec
#
Draft Report
Recommendation
Action
Official
Corrective
Action
Due
Date
2-8 Establish and implement minimum training
requirements for staff in all EPA programs
implementing or overseeing implementation of
data quality assurance activities.
AA/ORD
Non-concur as written. QAD will develop a
guidance document outlining a process to be used to
define training needs. It is the program managers'
responsibility to ensure that the needs within their
organization are defined and met.
3/99
2-9 Enhance oversight conducted by QAD by:
Not approving EPA organizations'
quality management plans which do
not include enough specifics to ensure
a sound quality assurance program.
i. Verifying that EPA organizations are
actually using the data quality
objectives process or an adequate
alternative systematic planning process
to identify environmental data needs.
:. Preparing and issuing timely
management assessment reports.
1. Tracking organizations'
implementation of corrective actions
identified in management assessment
reports.
Director, NCERQA
Director, NCERQA
Director, NCERQA
QAD
Quality Management Plans that are submitted for
Agency approval will be reviewed against the
criteria established in Chapter 3 of the Quality
Manual.
Management assessments will include explicit
review of the use and effectiveness of systematic
planning processes.
Draft assessment reports will be completed within
60 days of completion of data collection.
Corrective actions will be tracked using QA annual
reports and work plans. Where proposed schedules
are not being met, senior management will be
notified of the status.
Ongoing
Ongoing
Ongoing
Ongoing
-------
Rec
#
Draft Report
Recommendation
Action
Official
Corrective
Action
Due
Date
2-10
To elevate the visibility of quality assurance
progress in support of the Agency's stated
goal of sound science, prepare an annual
report to the Administrator on the "progress
and efficacy" in implementing the Agencywide
quality assurance program.
AA/ORD
Annual reports on QA program status will be
prepared for the Administrator.
1/98
3-1
Develop a strategy to encourage use of EPA's
7-step data quality objectives process as
EPA's recommended systematic planning
process as stated in the July 1998 EPA Quality
Manual.
QAD in conjunction with
OERR
QAD will work with OERR to develop a strategy to
enhance use of systematic planning, including the
data quality objectives process and alternative
processes for defining acceptance and performance
criteria.
3/99
4-8
Coordinate management assessments with the
national program managers to ensure that
regional staff satisfactorily perform their
quality assurance activities. At a minimum,
provide regional management assessment
results to the national program managers'
quality assurance managers.
Director, QAD
QAD will continue to involve QA managers from
related organizations in its management assessment
teams. Relevant results of regional management
assessments will be forwarded to national program
managers for their consideration.
Ongoing
4-9
Address the placement of regional quality
assurance managers in management
assessments to encourage regional
administrators to place quality assurance
managers at an organizational level where they
can be an effective advocate for the program.
Director, NCERQA
QAD will continue to evaluate the effectiveness and
the location and organizational support of QA
managers in the Regions as part of the management
assessments.
Ongoing
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
APPENDIX II
OSWER RESPONSE
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
office OP
SOLO WASTE AND EMERGENCY
RESPONSE
MEMORANpUM
SUBJECT: OSWER Response to OIG Draft Report on Superfund Quality Assurance
Program, Audit Report Wo. E1SKF7J
FROM-. TimoAhy Fields, Jr.
Acting Assistant Administra-
TO: Michael Simmons
Deputy Assistant Inspector tor internal Audits
Office of the Inspector General
Thank you for the opportunity to review the draft report on Superfund's Quality Assurance
Program. I appreciate the fact that you have been working closely with the Office of Emergency
and Remedial Response (OERR) staff on developing this report and have already incorporated
many of their comments in both the body of the text and in the recommendations. I will be
focusing our comments on the findings and recommendations for OSWER, presented in Chapters
3 through 5. Our comments are structured in the same format as they are presented in the draft
report. We note the Section on which we are commenting and the specific statements with
which we have concerns, and then follow with OSWER's comments. We established this
format, since OIG did not number the Sections in each of the Chapters. For the final report, we
recommend you establish a numbering system.
Chapter j - Findings,
Opening Paragraph and Opening Par^jiaph for Section Entitled-. MANAGERS HAD NOT
PROVIDED ADEQUATp DATA QUALITY OBJECTIVES DIRECTION OR TOOLS
"The policy was only minimally successful in the Superfund program because QAD and OERR
managers and Regional Administrators had not provided staff sufficient direction or tools to
implement EPA's data quality objectives policy."
This is an overstatement since OIG has not audited all of the Regions, but focused on Regions 3.
8, 9 and 10. It would be appropriate to qualify this statement as: In some Regions, this policy
was only minimally successful. Furthermore, OERR did provide some direction and tools to the
IMcycMfftecycuu* • PiWM wan V«gM»> 0» BJWM Into on 100% FUcycM <>»•€ (40% Poocomumerl
66 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
Regions - both the Superfund Data Quality Objectives guidance and QAD's Data Quality
Objectives Decision Error Feasibility Trials (DEFT) software were distributed to Regional staff.
Managers Had Not Provided Adequate Direction
"Also, Region 9 collected and analyzed about 420 samples that were unnecessary for decision
making." I am concerned that this misstatement remains in the report. As staff in OERR
previously indicated to you, these analyses were vital to ascertain the performance of a
wastewater treatment system used as part of a Superfund site cleanup. These samples were
necessary for overall site management and decision making.
"Superfund managers had not stressed that data quality objectives were required and that the EPA
7-step process was OERR's adopted approach." It is an overstatement to indicate that Superfund
managers had not stressed data quality objectives; you need to qualify this statement by adding, In
some Regions, to the beginning of the sentence. Since OERR also wants to ensure Regional
flexibility, EPA's 7-step process is one example of an approach that can be followed. However,
we do agree that the Regions need to be aware of this flexibility as you state later in that
paragraph.
Chapter 3 - Recommendations
Recommendation 3-2
"In concert with QAD, develop and implement a plan to institutionalize the Superfund
program's required data quality objectives process."
OSWER agrees with this recommendation. OERR will work with QAD to develop a plan by
March 31, 1999.
Recommendation 3-3
"Advocate the benefits of developing data quality objectives using an across-organization team
approach."
OSWER agrees with this recommendation. OSWER will provide this recommendation to the
Regions in a Quality Assurance (QA) memorandum from the Acting Assistant Administrator,
OSWER to the Regional Administrators and Superfund Division Directors by December 31,
1998.
Recommendation 3-4
"Provide Superfund staff with sufficient tools to implement EPA's data quality objectives policy in
the Superfund program by:
a) "Making data quality objectives training more effective by integrating available quality
67 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
assurance courses to provide a comprehensive understanding of data quality objectives."
OSWER agrees with this recommendation. This work will be conducted in concert with
Recommendation 3-2 and will be completed by March 31, 1999.
b) "Making automated tools (such as the Data Quality Objectives Decision Error Feasibility Trials
software or other interactive modeling software) available to appropriate staff and encouraging
use through training and examples of success."
As previously noted, OERR has completed this task. However, OERR will request QAD to
provide the software again by January 31, 1999. OSWER will inform Regional management of
the availability of the software in OSWER's QA letter to the Regions as discussed in
Recommendation 3-3.
c) "Making more expertise available to provide staff technical assistance in areas such as statistics
either through realignment of regional resources or contractor support."
As the recommendation is currently written, OSWER does not agree with it. It is implying that
OERR provide additional resources for Regional quality assurance responsibilities. As you know,
OERR is currently addressing many competing priorities and at this time does not project that this
can be accomplished. However, if the recommendation is rewritten as, Increase awareness that
expertise is available to provide staff technical assistance in areas such as statistics either through
realignment of Regional resources or contractor support,
OSWER will support it in the QA memo to the Regions expected by December 31, 1998.
d) "Sharing regionally-developed tools, procedures, best practices, and successes with all EPA
programs and regions."
As the recommendation is currently written, OSWER does not agree with it. It is not OSWER's
responsibility to reach out to all EPA programs and regions. However, if the recommendation is
rewritten as, Sharing regionally-developed tools, procedures, best practices, and successes with
QAD and Superfund staff\ OSWER can support it. QAD serves as the central point of contact on
QA issues for the Agency. For Superfund-specific issues, this can be accomplished by September
30, 1999 when OERR's Regional Quality Assurance reviews are completed, and OERR develops
guidance to address concerns and share successes revealed during the QA reviews.
Recommendation 3-5
"Recommend to each regional administrator that they review the most recent QAD, OERR, and
regional quality assurance management assessment results for their unit to determine whether the
regional administrator should identify quality assurance implementation as a material weakness in
their unit as part of the fiscal 1998 Federal Managers Financial Integrity process."
68 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
OSWER does not agree that quality assurance concerns should be raised to the level of a
material weakness, especially in light of the work that OERR is pursuing on quality assurance
activities (e.g., QA reviews, guidance, memorandums). Additionally, training is available to the
Regions from the Quality Assurance Division which hopefully will alleviate the problem in the
future. However, OSWER is willing to work with QAD on a memo to the Regional
Administrators requesting that they review the QA management assessment results to determine
whether a material weakness exists in their Region. QAD's proposed date for this memo is
September 30, 1998.
Chapter 4 -Findings
Opening Paragraph
"Top OERR and regional managers had not ensured staff adequately performed Superfund quality
assurance oversight...."
OSWER is concerned with the absolute nature of this statement and would prefer it be qualified
to read: Regional managers had not uniformly ensured staff adequately performed Superfund
quality assurance oversight. It is the responsibility of Regional managers, not OERR, to oversee
day-to-day operations in the Regions.
SENIOR SUPERFUND MANAGERS HAD NOT IDENTIFIED OVERSIGHT
WEAKNESSES
The first paragraph and the title of this section contain numerous absolute statements that need to
be qualified. The title should be modified to reflect that Senior Superfund managers had not
consistently identified oversight weaknesses. This continues for the rest of the paragraph; as we
stated earlier, the OIG audit did not cover all ten Regions, nor was every file in each of the
Regions reviewed; therefore these statements need to be qualified.
OERR Had Not Fulfilled Its Quality Assurance Oversight Responsibilities
Again, we would like to see the subtitle include a qualifier such as, OERR had not completely
fulfilled its Quality Assurance Oversight Responsibilities.
The statement that "Removals have minimal quality assurance requirements up-front due to the
time-critical nature of the activity," is not accurate. As a reminder, the removal program also
includes non-time critical actions, and these longer term cleanups do follow QA requirements
up-front.
Regional Administrators Had Not Ensured Previously Identified Oversight Weaknesses Were
Corrected
69 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
Again, we would like to see this subtitle qualified since OIG observations are based on limited
examples in a limited number of Regions.
SUPERFUND MANAGERS HAD NOT DEMONSTRATED QUALITY ASSURANCE
OVERSIGHT COMMITMENT.
OSWER continues to be concerned with the above phrase. As previously noted, OERR
recognizes the importance of data quality assurance and the necessity of management involvement
in such issues. Stephen Luftig, Director, OERR, Larry Reed, Deputy Director, OERR, and Dana
Tulis, Director, Analytical Operations and Data Quality Center, will continue to be involved in
OERR's quality assurance program. Duane Geuder and Joan Fisk, both of whom have been
involved in quality issues for Superfund for many years, have committed a large percentage of
their time to working towards the kind of improvements to our Superfund Quality System that
have been identified as problem areas by the OIG.
Furthermore, I established the Field and Analytical Services Teaming Advisory Committee
(FASTAC) in my March 16, 1998 memorandum, Sources for Analytical Services Under
Contracts 2000. FASTAC is comprised of Headquarters, Regional Superfund, and Regional
Science and Technology managers who are charged to address field and analytical issues that cut
across the Superfund program and analytical support organizations. Quality Assurance is one of
the implementation issues being addressed by FASTAC.
Organizational Location of Quality Assurance Managers Undermined Their Authority
As previously discussed, OSWER believes the Superfund Quality Assurance Manager is
working at the appropriate organizational level. When OERR reorganized three years ago, the
Analytical Operations/Data Quality Center (AOC) was designed to include oversight and
implementation of quality assurance activities for both CLP and non-CLP data. AOC has and will
continue to provide independent technical assessments for analytical services outside of the CLP
program. Furthermore, FASTAC is addressing the Superfund Quality Assurance program, with
emphasis on planning activities. Larry Reed is the FASTAC Chair and Dana Tulis has the lead for
FASTAC implementation. OERR is also establishing quality assurance leads in each of OERR's
Regional Centers.
Chapter 4 - Recommendations
Recommendation 4-1
"Develop a plan with a realistic cycle for OERR staff to perform management and technical
assessments in the regions. Include in the plan the staffing resources necessary to carry out the
plan. Invite regional Superfund and quality assurance managers to participate as members of
70 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
OERR's assessment team."
OSWER agrees with the recommendation and the plan will be complete by December 31, 1998.
Recommendation 4-2
"Consider joint management assessments with QAD to ensure that regional Superfund staff
satisfactorily perform their quality assurance activities. At a minimum, obtain QAD regional
management assessment results."
OSWER agrees with this recommendation. OERR just completed a joint assessment with QAD
for the Region 10 review. In the past, OERR's Quality Assurance manager has participated in
QAD reviews as well.
Recommendation 4-3
"Work with regional Superfund staff to develop adequate oversight procedures to ensure data is
of sufficient quality to support decision making."
OSWER agrees with this recommendation. OERR will be developing such guidance once the
Regional QA reviews are complete. The guidance, as previously noted, should be completed by
September 30, 1999. OERR will also work with Regional QA staff. The Recommendation
should probably be broadened to include Regional QA staff as well.
Recommendation 4-4
"Develop a strategy to ensure corrective actions in response to OIG audits and EPA management
assessments are fully implemented."
OSWER agrees with this recommendation. OSWER will request a strategy from each of the
Regions in the QA memo that I will be issuing to the Regions by December 31, 1998.
The regional plans will be requested by February 28, 1999.
Recommendation 4-5
"Develop guidance to effectively implement the Superfund quality assurance program including
specific quality assurance responsibilities for regional project managers."
OSWER agrees with this recommendation. This will be addressed by OERR's QA guidance
expected by September 30, 1999.
Recommendations 4-6 and 4-7
"Place the Superfund quality assurance manager and staff at an organizational level where they
can be an effective advocate for the program." "Clearly and formally delegate program authority
to the headquarters Superfund quality assurance manager."
71 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
OSWER believes this is already the case. The Superfund Quality Assurance Manager does not
directly report to AOC, which is responsible for operations of the QA program. The Quality
Assurance Manager has direct access to OERR's Office Director and Deputy Directors, and
routinely reports to the Deputy Director responsible for QA issues. All memorandums concerning
improvements and modifications to the QA program will be disseminated under OERR's Office
Director or when appropriate, under my signature.
Chapter 5 - Findings
Please ensure each of the subtitles in this section are appropriately qualified since all the Regions
were not audited. "OERR and Regional Administrators had not identified and provided necessary
training," should only refer to some Regional Administrators. Similarly, some "Senior managers
were not held accountable and did not demonstrate their commitment to Superfund quality
assurance training."
Chapter 5 - Recommendations
Recommendation 5-1
"Implement quality assurance training for environmental data operations." This
recommendation consists of seven subtasks.
OSWER supports the overall recommendation; however, we do not agree that OERR is the lead
on each of the subtasks. The overall recommendation can be implemented by September 30,
1999.
a) "Identifies the skills necessary to perform Superfund-specific quality assurance responsibilities."
OERR will work with QAD to establish these in concert with the Regions.
b) "Establishes quality assurance performance language for appropriate staff."
If QAD provides these, OERR will share them with their Regional managers.
c) "Assesses each staff member's quality assurance skills with respect to established standards."
This is clearly a Regional management responsibility.
d) "Identifies minimum Superfund quality assurance training requirements for staff to perform
their responsibilities and includes the requirements in regional quality management plans."
OERR can work with QAD in establishing training requirements. The Regional managers will
need to include the requirements in the regional quality management plans.
72 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
e) "Develops and delivers an effective quality assurance training program that meets minimum
mandatory Agency wide requirements and incorporates these requirements into minimum
Superfund training requirements."
OERR will work with QAD.
f) "Ensures Superfund staff needing quality assurance training complete required training."
Regional management will need to ensure that staff needing quality assurance training complete
required training.
g) "Assesses and modifies training based on participants' course evaluations."
OERR will work with QAD and the Regions.
Recommendation 5-2
"Assist regions in providing the necessary training when regional training does not prove
entirely effective."
OSWER agrees with this recommendation and can provide assistance as resources allow.
Recommendation 5-3
"Provide senior managers with sufficient training for them to understand the importance and
necessity of the quality assurance program."
OERR will provide this recommendation to the Regions and request it be accomplished by
September 30, 1999.
Thank you for incorporating our comments. We look forward to reviewing the final report.
cc: Stephen Luftig, OERR
Larry Reed, OERR
Dana S. Tulis, OERR
Henry Longest, ORD
Nancy Wentworth, ORD
Superfund National Policy Managers
Regional RS&T Division Directors
Bob Cianciarulo, Region 1
Elizabeth Harris
73 Report No. E1SKF7-08-0011-8100240
-------
EPA Had Not Effectively Implemented
Its Superfund Quality Assurance
Program
APPENDIX II
DISTRIBUTION
Office of Inspector General
Inspector General (2410)
Deputy Assistant Inspector General for
Internal Audits (2421)
Headquarters Audit Liaison (2421)
Divisional Inspectors General for Audit
Headquarters Offices
Assistant Administrator for Solid Waste and Emergency Response (5101)
Director, Office of Emergency and Remedial Response (5201)
Division Director, Analytical Operations and Data Quality Center (5202G)
Division Director, Regions 5/7 Accelerated Response Center (5202G)
Assistant Administrator for Research and Development (8101)
Director, National Center for Environmental Research and Quality Assurance (8701)
Division Director, Quality Assurance Division (8724R)
Agency Followup Official (2710)
Agency Followup Coordinator (3304)
Associate Administrator for Regional Operations and State/Local Relations (1501)
Associate Administrator for Congressional and Legislative Affairs (1301)
Associate Administrator for Communications, Education, and Public Affairs (1701)
Regional Offices
Regional Administrators (RA)
74 Report No. E1SKF7-08-0011-8100240
------- |