-------
d) The use of the peer-review process, which has undoubtedly contributed to the
quality of its guidance and requirement documents.
The Subcommittee also found issues that require the attention of the Agency's
management such as:
a) The need to include all activities, which have the potential to affect the quality of
the Agency's products and services, under the auspices of the Quality System;
b) The need for Agency management to review the appropriateness of the reporting
status of the Quality System function within the Agency organization. The
Subcommittee recognizes that this recommendation is of a policy nature.
However, the Subcommittee believes this recommendation is justified due to the
impact of reporting status on the efficacy of the Quality System;
c) The lack of an Agency-wide focal point for quality issues and needed corrective
actions;
d) The need for Quality System training to be expanded to include the training of
senior management;
e) The need to identify metrics for "bench marking" existing levels of quality and
changes over time;
f) The need for guidance appropriate for the development and use of
mathematical/computer models and the associated data; and
g) The need to determine if budgeted resources for QAD are sufficient to meet the
.*. increased demand for its services.
These findings and others are discussed in more detail on the following pages.
The QM review consists of three parts. The first part addresses the first two levels of the
EPA's Quality System (Policy & Organization). The second part examines the third level of the
Agency's Quality System (Project). The first two parts were addressed during the April meeting
and are summarized in this letter. The third part of the review, which is tentatively scheduled for
September 1998, will consider implementation of the Quality System. A more detailed charge is
presented in Appendix A.
-------
As part of the review, QAD and its management gave a detailed description of its past,
present and planned activities. The Subcommittee was impressed with the high-level of
professionalism, knowledge and obvious dedication of the QAD Staff, yet not surprised in light
of the significant and positive impact this small group of scientists has had not only across the
Agency but across other Federal agencies and the regulated community.
1. Part 1 Policy & Organizational Component Levels of the Agency's Quality System
The first part of the review focused on relevance, completeness and practicality of the
Agency's quality policy as defined in EPA Order 5360.1, the Quality Manual and the
organizational components designed for policy implementation as described by the Agency's
Quality System (EPA QA/G-0).
a) Relevance
(1) The Subcommittee found the proposed revision to Order 5360.1 and the
requirements in the Quality Manual to be relevant to the achievement of
Agency goals. These documents establish a quality assurance (QA)
management structure which encourages Agency personnel and recipients
of EPA funding to collect data of known and sufficient quality adequate to
support the decisions for which the data were collected.
(2) The Subcommittee found the existing organizational component guidance
(EPA QA/R-2 - Quality Management Plans and EPA QA/G-3 -
Management System Reviews) to be relevant to the achievement of
Agency goals and to the control of certain activities specified by the
Quality Manual. Documentation for the training component is not yet
available, but as outlined during the review by QAD personnel and as
implemented in QAD's existing training programs, the training component
should be relevant to the successful implementation of the Quality System.
(3) The Subcommittee recommends that EPA periodically review the policy
and associated organizational components to determine whether they
remain relevant and, if necessary, revise them. QAD staff noted that
documents are now issued with a five year "sunset" provision which
requires documents to be either reissued, revised or revoked.
(4) The Agency's mission statement contains two responsibilities, protection
of human health and safeguarding the environment, which are not clearly
-------
reflected in EPA Order 5360.1. While the Agency's work culture and
practices may include "health" under "environment", the outside reader
has no way of understanding this integration from the Agency's Quality
System documentation. Because health protection is a major
responsibility of the Agency, the Subcommittee recommends that the
inclusion of health data in the Quality System be clearly stated and visible
throughout all quality system documentation.
b) Completeness
(1) The overall success of the Agency's Quality System will depend upon
oversight of all pertinent activities with the potential to affect the quality
of the Agency's services and products. Agency policy (as defined by
5360.1) and the Quality Manual focus on environmental measurement
programs. Neither document identifies the universe of Agency activities
that need to be monitored under the Agency's quality system. This may
result in spotty coverage of quality issues.
The Subcommittee recommends that the Agency: (a) identify all Agency
services and products; (b) identify all activities critical to the quality of
these services, products and the achievement of Agency goals; © identify
which of these activities are presently subject to the auspices of the
Agency's Quality System; and (d) identify any remaining activities that are
not presently covered by the Quality System (e.g., models) and modify the
Policy and requirements to incorporate the activities under the auspices of
the Quality System. (Refer to Figure 1 for a graphical depiction of
covered and non-covered activities.)
(2) The Quality System documentation lacks an organizational chart detailing
the formal authority, responsibility, reporting and communication lines
that must be understood across the Agency to ensure success of the
Quality System. The Subcommittee recommends that the role of the QAD
and any other organization responsible for the Quality System be
explicitly portrayed in Agency documents such as the Quality Manual and
EPA QA/G-0.
(3) The Subcommittee recommends that the Quality System policy require
development of metrics for "bench marking". Qualitative, and where
possible quantitative, metrics will establish existing levels of quality and
-------
identify changes in quality over time. Such metrics could be used across
the Agency to measure understanding of, and commitment to, the Quality
System.
(4) The quality system organization/program appears to be conceptually
complete. One key organizational component, the training guidance, is
not yet final and was not reviewed. The Subcommittee suggests that QAD
develop a comprehensive training strategy with effective target audience,
needs assessment and evaluation components to ensure the ongoing
success of the training program. Regarding the presently incomplete
training guidance, the Subcommittee recommends that, when complete,
the training documentation address the training of senior managers since
their understanding of the Quality System and their role are key to its
success.
(5) The Subcommittee recommends that QAD update its mission statement
and clearly identify its customers. The Subcommittee believes this will
allow QAD to appropriately allocate its resources as demand for its
services increases as well as make QAD's role more clearly understood
within and outside of the Agency.
c) Practicality
(1) The success of the Quality System is, in part, dependent upon access to the
proper level of authority by those who lead the Q A management efforts.
The Subcommittee believes that the Agency's organization does not
appropriately accommodate this Quality Management need, since the first
presence of a Quality Management function within the Agency occurs at
the 4th or 5th tier. The Subcommittee recommends that: (a) the Agency
identify a lead Quality Management organization; (b) incorporate the
Quality Management organization at an elevated position within the
Agency's organization such that it would have the appropriate level of
authority; and © define the senior management position for the Quality
Management organization such that it would be free from the turnover of
appointed positions. The Subcommittee recognizes that this
recommendation is of a policy nature. However, the Subcommittee
believes this recommendation is justified due to the impact of reporting
status on the efficacy of the Quality System.
-------
(2) The Policy does not identify an Agency-wide focal point for quality
issues and needed corrective actions that could be approached by both
those internal and external to the Agency. For example, during its review,
the Subcommittee examined a letter, which alleged that proposed
regulatory standards may have been based on poor-quality data. After
reviewing this letter, the Subcommittee could not identify an Agency
organization having the authority to investigate such quality issues and
initiate corrective actions.
(3) Although the revised EPA Order 5360.1 designates QAD as the "central
management authority", section 7.a.(l) limits this authority to four
requirements the development of training, management assessments, QMP
review and approval and review and revision of Quality System policy.
As a consequence, QAD's role is ambiguous and needs to be clearly
defined.
(4) The external pressures for increased quality, growing acceptance of
existing QAD requirements and guidance, and the planned release of new
guidance documents will increase demands on the already oversubscribed
QAD staff. Due to QAD's critical role in the Agency's Quality System,
the Subcommittee recommends that the Agency consider whether
sufficient resources are available. Among other critical responsibilities,
QAD must support the Performance Based Measurement System (PBMS),
use of secondary data, training and management system reviews.
(5) The QAD implemented Management System Reviews (MSRs) have
offered both a front line perspective and a metric of how the Quality
System has been implemented. MSRs have detected inconsistent
implementation across the Agency's regions and program offices and have
also uncovered issues that influence successful implementation (e.g., the
management levels to which Quality Assurance Managers (QAMs)
report). The Subcommittee recommends that the Agency investigate
whether more frequent use of MSRs would facilitate acceptance of the
Quality System.
(6) The policy specified reporting structure and authority for quality
assurance managers (QAM) are vague. QAMs are most effective when
they report to the highest career professional in an organization. EPA
-------
does not appear to apply this finding to policy or practice. To improve the
overall Quality System, the Subcommittee recommends that EPA rewrite
the Quality Management Plan (QMP) requirements document (EPA
QA/R-2) to reflect this and other findings (e.g., the need for QAM backup
to prevent disruptions resulting from QAM travel or vacations).
2. Part 2 Review of the relevance, completeness and practicality of the Agency's
project-level requirements and guidance for data collection activities
The second part of the review focused on the relevance, completeness, and practicality of
the "project" level of the Agency-wide Quality System as defined by existing requirement and
guidance documents (G-4, R-5, G-5, G-6 and G-9) and future activities and planned documents
as described by the QAD staff.
a) Relevance
(1) The Subcommittee found the existing project-level requirements and
guidance (G-4, R-5, G-5, G-6 and G-9) and the planned guidance (G-7 and
G-8) to be relevant to the achievement of Agency goals. These documents
establish project level guidance that encourages Agency personnel and
recipients of EPA funding (including contractors, researchers, state and
local agencies, and Tribes) to collect data of known and sufficient quality
adequate to support the decisions for which the data were collected.
(2) The Subcommittee encourages QAD to continue its plans to develop
program and matrix specific guidance such as QA-G4HW. The pedagogic
value of these guidance documents will highlight the relevance of the
planning process to specific programs and facilitate acceptance of the
Quality System.
b) Completeness
(1) The existing and planned project-level requirements and guidance provide
a comprehensive blueprint for the planning, implementation and
assessment of the data collection operation.
(2) According to Agency policy, the Quality System covers data collection
activities and the design, construction and operation of environmental
-------
technology. Pertinent guidance does not now exist for technology. QAD
plans to develop guidance for environmental technology (G-12) based on
existing US Army Corps of Engineers guidance. The Subcommittee
agrees that this is a logical approach to the development of guidance and
urges QAD to obtain peer review of this guidance document as was done
with the previous documents.
(3) The Agency's plans for defining quality system requirements and guidance
for the review and use of mathematical and computer models are unclear.
Over time the Agency has increased the use of models to "generate" data
and in some cases the results are used for regulatory purposes. In 1992
the Agency Task Force on Environmental Regulatory Modeling
(ATFERM) developed guidance for conducting external peer review of
regulatory models.
Recently, EPA hosted the Models 2000 conference and established
an internal Committee on Regulatory Environmental Modeling (CREM)
under the direction of the Agency's Science Policy Council (SPC).
The SAB also finds modeling to be an important issue. The SAB
has long been interested in the scientifically appropriate use of models at
EPA. In 1989 the SAB issued the Resolution on the Use of Mathematical
Models by EPA For Regulatory Assessment and Decision-making
(EPA-SAB-EEC-89-012), in 1993 it reviewed Draft Agency Guidance for
conducting External Peer Review of Environmental Regulatory Modeling
(EPA-SAB-EEC-LTR-93-008), and in the last five years the SAB has
reviewed several models for EPA. SAB members participated in the
Models 2000 workshop and the Environmental Models Subcommittee
(EMS) of the SAB's Executive Committee is presently investigating the
Agency-wide use of models and the use of data generated from the
models.
At this time, QAD is unsure about how to incorporate models into
the Quality System. The Subcommittee recommends that models and
model-generated data be covered by the Agency's Quality System in
accordance with the Quality Manual (page 1-3 Section 1.3.1) and the new
draft order 5360.1 (p. 3 Section 5b(3)) and in a manner, that is specific and
appropriate for computer/mathematical models and associated data. In
-------
developing guidance for models and model-generated data, QAD may
wish to consider the ATFERM and existing SAB guidance on model use
and peer review. Such documents may help because they were written
with the intent of having models provide "quality" information. QAD will
also find it helpful to consider the activities and reports of the SAB's new
Environmental Models Subcommittee.
(4) The Subcommittee recommends that guidance for Step 3 of the Data
Quality Objective (DQO) planning process be rewritten to address the
problem of linking EPA-generated data bases with externally generated
data bases. For example, answering many health-related questions
frequently requires linking environmental data with census, cancer or birth
defect registry data, or other data systems (such as water distribution
maps) to determine whether there is a relationship between the
environmental measurements and health. Without identifying some
mutually-agreed linking variable (perhaps latitude-longitude, block
number, etc.) or strategy, critical questions cannot be answered regarding
health, risk assessment and standard-setting. It is imperative that the
linking issue be addressed in the planning stage so all parties will be
aware of what needs to be done or can't be done to link data systems. This
early consideration will reduce the probability of later frustration and
dissatisfaction.
(5) The intent of the DQO planning process is to be inclusive regarding key
stakeholders. The Subcommittee recommends that the next revision of
Step 1 of the G-4 guidance acknowledge that many projects will have
multiple stakeholders. As a result many questions need to be addressed
collectively and when possible consensus built before data are collected.
The revised guidance needs to assist staff in identifying the appropriate
range of stakeholders, discuss the potential need to obtain acceptable
answers to more than one question and discuss how quality is not
bargained away by proceeding with a consensus-based plan.
c) Practicality
(1) The Subcommittee finds that the structural form of QAD's project-level
guidance is an appropriate and practical consequence of addressing the
policy and program requirements set forth in ANSI/ASQC E-4.
-------
(2) The Subcommittee finds the structured planning process, the detailed
documentation of its outputs in a Quality Assurance Project Plans
(QAPP), and the mandatory requirement that all QAPPs be approved by
the Agency prior to project initiation provides confidence that all pertinent
data collection issues have been addressed.
(3) The Subcommittee recommends that revised guidance require that, where
the DQO process was not employed, the QAPP discuss how the objectives
were identified.
(4) QAD has been a leader in defining terminology and thus improving
communications about data collection activities. The Subcommittee
suggests two areas for further improvement:
(a) First, due to the increasingly multi-disciplinarian audience for QAD
documents, it would be helpful, when possible, to employ more
universally understood terms or in those many cases when a term having
more than one meaning (e.g., population, stakeholder, sample) must be
employed that it be defined within the document.
(b) Second, minimize the unnecessary use of synonymous terms so that
the reader will not incorrectly assume additional levels of complexity (For
example, the trail from DQOs to the documentation of Data Quality
Indicators in a QAPP is littered with terms such as quality objectives,
quality standards, quality criteria, criteria for measurement criteria, quality
control, quality control requirements, QC procedures, acceptance criteria,
measurement performance criteria and PARCC.).
The, Subcommittee commends the NCERQA and QAD management for their
commitment to quality improvement as evidenced by their request for a review of the
Agency-wide Quality Management Program. The Subcommittee recognizes the importance of
the Agency's Quality Management Program and QAD's role in developing the Quality System to
its present sfete. The Subcommittee believes the Agency has benefitted greatly from the
dedicated efforts of the QAD staff and that further benefit could be obtained by increasing
support for more of the same type of effort.
The above findings and recommendations reflect an appreciation for the significant
contributions made towards the quality of data collection activities while recognizing that the job
10
-------
is not nearly done. Continued and increased attention from senior management is needed to
implement the Quality System uniformly to all activities on an Agency-wide basis.
We look forward to your response to the advice contained in this report.
Sincerely,
Dr. John Maney,
Quality Management Subcommittee
Environmental Engineering Committee
Dr. Hilary Inyang, Chair
Environmental Engineering Committee
Science Advisory Board
. Joan Daisey, Chair
Science Advisory Board
11
-------
Figure 1. Quality System and Agency Activities
-,i ,r------
POLICY
Administrator
Assistant Administrators
Policy
Order 5360.1,
Quality Manual
i
ORGANIZATION
Programs/Regions
QMP, MSR, | I
QAARWP . I
l«
I'
PROJECTS
DQO, QAPP, 1 I
SOP.TA, DQA , |
Project Products & Services
|0
co .a
!_.«
ffi W H
sŁ|
*
5^1
I I
1 »a
C -5
0
1 I
Ł S
Aoencv Products &
Services
Including Regulatory/Multl-
Media/Multi-disclpllnary/Value
decisions based on multi-
project input
i ss*
o Q.T5
< w 3
>» o O
z >
e o
c «
-------
APPENDIX A - Proposed Charge for QM Review
The QM review consists of three parts. The first part addresses the first two levels of the
EPA's Quality System (Policy & Organization). The second part examines the third level of the
Agency's Quality System (Project). (Refer to Figure 0-2 of EPA QA/G-0 for a depiction of the 3
Quality System levels and Section 1.3 of the Quality Manual for covered activities.) The third
part of the review will consider implementation of the Quality System. This review also ties the
QM review in with EPAs mission statement and goals, a driver that should be behind all Agency
activities.
1. A Proposed Charge for Part I of the Quality Management Subcommittee Review ~ To
evaluate the relevance, completeness, and practicality of the "policy" and
"organizational" components of the Agency-wide Quality System.
a) Relevance: Evaluate whether the "policy" and "organizational" components of the
Agency's Quality System are applicable to the Agency's mission statement, goals
(EPA/190-R-97-002) and those activities identified in Section 1.3 of the Quality
Manual.
b) Completeness: Evaluate whether the "policy" and "organizational" components of
the Agency's Quality System are structured such that the quality of all Agency
activities needed to comply with the Agency's mission statement and goals will be
monitored and assessed versus performance measures.
c) Practicality: Is the present structure of the "policy" and "organizational"
components of the Agency's Quality System designed for success (i.e., will the
policy and organizational levels of the Quality System properly assess and control
* pertinent activities and facilitate achievement of EPA goals).
2. A Proposed Charge for Part II of the Quality Management Subcommittee Review ~ To
evaluate the relevance, completeness, and practicality of the "project" level of the
Agency-wide Quality System.
a) Relevance: Evaluate whether the "project" level of the Agency's Quality System
is applicable to the Agency's mission statement and goals (EPA/190-R-97-002)
and the covered activities identified in Section 1.3 of the Quality manual.
b) Completeness: Evaluate whether the "project" level of the Agency's Quality
System is structured such that of all project activities needed to comply with the
A- 1
-------
Agency's mission statement and goals will be monitored and assessed versus
performance measures. Evaluate whether the project level guidance documents
consider all essential aspects necessary to monitor and measure the quality of
environmental measurement data.
c) Practicality: Is the present structure of the "project" level of the Agency's Quality
System designed for success (e.g., are they cost-effective, efficient to implement,
understandable by the intended audience, will the project level of the Quality
System facilitate achievement of EPA goals).
3. A proposed charge for Part III of the Quality Management Subcommittee Review -- To
use available information to evaluate the Agency's success in implementing the
Agency-wide Quality System.
A-2
-------
U.S. ENVIRONMENTAL PROTECTION AGENCY
Science Advisory Board
Environmental Engineering Committee
Quality Management Subcommittee
July 21,1998 Meeting
CHAIR
Dr. John P. Maney, President, Environmental Measurement Assessment, Hamilton, MA
MEMBERS
Dr. Edgar Berkey, Vice President and Chief Science Officer, Concurrent Technologies Corporation,
Pittsburgh, PA
* Dr. Hilary I. Inyang, University Professor and Director, Center for Environmental Engineering,
Science and Technology (CEEST), University of Massachusetts Lowell, Lowell, MA
Dr. JoAnn Slama Lighty, Associate Dean for Academic Affairs, Associate Professor of Chemical
Engineering, University of Utah, College of Engineering, Salt Lake City, UT
MEMBER OF OTHER SAB COMMITTEES
* Dr. William J. Adams, Director, Environmental Science, Kennecott Utah Copper Corp., Magna, UT
(Ecological Processes and Effects Committee)
CONSULTANTS
Dr. Mohammad A. Ansari, President, American Institute for Pollution Prevention, Chester, VA
Dr. Gordon KJngsley, School of Public Policy, Georgia Institute of Technology, Atlanta, GA
* Dr. Michael J. McFarland, River Heights, UT
Dr. Rebecca-Parkin, Director, Scientific, Professional, and Section Affairs, American Public Health
Association, Washington, DC
Dr. Douglas Splitstone, Splitstone & Associates, Murrysville, PA
x
* Did not attend meeting
SCIENCE ADVISORY BOARD STAFF
Kathleen W. Conway, Designated Federal Officer, U.S. EPA, Science Advisory Board (1400), 401 M
Street, SW, Washington, D.C. 20460
Dorothy M. Clark, Management Assistant, U.S. EPA, Science Advisory Board (1400), 401 M Street,
SW, Washington, D.C. 20460
-------
NOTICE
This report has been written as part of the activities of the Science Advisory Board, a
public advisory group providing extramural scientific information and advice to the
Administrator and other officials of the Environmental Protection Agency, The Board is
structured to provide balanced, expert assessment of scientific matters related to problems facing
the Agency. This report has not been reviewed for approval by the Agency and, hence, the
contents of this report do not necessarily represent the views and policies of the Environmental
Protection Agency, nor of other agencies in the Executive Branch of the Federal government, nor
does mention of trade names or commercial products constitute a recommendation for use.
-------
DISTRIBUTION LIST
Administrator
Deputy Administrator
Assistant Administrators
EPA Regional Administrators
Director, Office of Science Policy, ORD
EPA Laboratory Directors
EPA Headquarters Library
EPA Regional Libraries
EPA Laboratory Libraries
Library of Congress
National Technical Information Service
Congressional Research Service
------- |