12TH ANNUAL NATIONAL MEETING ON
MANAGING ENVIRONMENTAL DATA QUALITY
R O
D
Albuquerque, New Mexico February 10-14,1992
Quality Assurance Management Staff
U.S. Environmental Protection Agency
-------
EPA'S MANAGEMENT SYSTEM FOR
ENVIRONMENTAL DATA QUALITY
TOTAL QUALITY MANAGEMENT (TQM) is the process whereby an
organization, led by senior management, commits to focusing on
quality as a first priority in every activity. TQM implementation
creates a culture in which everyone in the organization shares the
responsibility for improving the quality of products and services,
and for "doing the right thing, the right way, the first time."
EPA's QUALITY ASSURANCE (QA) program for environmental data
operations is based firmly on the principles of Total Quality
Management. Quality assurance is the process of management review
and oversight at the planning, implementation, and completion
stages of an environmental data operation to assure that the data
provided by a line operation to data users are of the quality
needed and claimed. The TQM concepts which the Agency's QA program
has put into practice include the following:
* Customer-supplier relationships, especially a clear
statement of the customer's (data user's) needs;
* Establishment of measures of performance for supplier
implementation, and customer evaluation;
* Process analysis through techniques such as process flow
diagramming; and
* Employee development, involvement, and recognition.
QA is not identical to QUALITY CONTROL (QC), which is an
aspect of the implementation phase of an environmental data
operation. QC includes those activities required during data
collection to produce the data quality desired and to document the
quality of the collected data (e.g., sample spikes and blanks).
At EPA, quality assurance is a management system based upon
the proven management philosophy of Total Quality Management. The
primary responsibility for implementing QA belongs to the line
managers of EPA organizations which are involved in the collection
or use of environmental data, whether in Headquarters, Regions, or
Research and Development Laboratories. EPA managers at all levels
benefit from a program which succeeds in bringing the Agency's
environmental data operations into alignment with its decision-
making needs: "the right thing, the right way, the first time."
-------
CONTENTS
Agenda i
Meeting Highlights iii
Speaker Biographies v
INTRODUCTION
Introductory Speakers 1
PRESENTATIONS
Keynote Address by Eugene Rouleau 3
EPA's Environmental Monitoring Management Council
by Fred Haeberer 7
ANSI/ASQC E4, The Quality Assurance Requirements for QA Systems
by Gary Johnson 11
Agency-Wide Data Standards by Steve Hufford 13
The News From QAMS by Nancy Wentworth 15
Total Quality Management: What Does It Mean to the Analytical
Laboratory? by Jackson Hicks 17
PANEL DISCUSSION
Merging Quality Into Science and Research 21
DATA QUALITY ASSESSMENT
Research Triangle Institute's Support to QAMS7 Current
Activities and Ongoing Research by Daniel Michael 31
QAMS' Vision by Fred Haeberer 35
Assessment of Data Usability in Superfund by Larry Reed 37
Assessing Data Quality For Agency Pesticide Decisions
by Richard Schmitt 39
Assessment of Error in Soil Data by Jeff van Ee 43
Using Power Analysis to Assess Office of Water Data Quality
by Robert Graebner 45
QUALITY ASSURANCE MANAGER OF THE YEAR AWARD
Banquet Presentation by Ralph Bauer. 53
Acceptance Speech by Kendall Young 61
TRAINING SESSIONS
The Training Doctor Is In 63
Career Skills and Strategy for Marketing QA 63
Essential Skills for Change Agents 63
Environmental Statistics for the QA Practitioner 64
GROUP SESSIONS
Office of Research and Development 65
National Program Offices 69
Regional Program Offices 71
Joint Session 73
-------
AGENDA
Monday, February 10
1:00 pm - Introduction
Nancy Wentworth, Director, Quality Assurance Management Staff
1:10 pm - Welcoming remarks
Louis E. Saavedra, Mayor of Albuquerque
1:20 pm - Keynote Address
Eugene Rouleau, Deputy Regional Director, U.S. Office of Personnel
Management
3:00 pm - EPA's Environmental Monitoring Management Council (EMMC)
Fred Haeberer, Quality Assurance Management Staff
3:30 pm - The ANSI/ASQC E4 19xx: Quality Assurance Program
Requirements for Environmental Programs
Gary Johnson, Quality Assurance Management Staff
3:40 pm - Agency-Wide Data Standards
Stephen Hufford, Information Management Branch, Office of
Information Resources Management
4:10 pm - The News From QAMS
Nancy Wentworth, Director, Quality Assurance Management Staff
Tuesday, February 11
8:30 am - Merging Quality into Science and Research (panel
discussion)
Gary Johnson, Quality Assurance Management Staff (CHAIR)
Lindsey Wood, Proctor & Gamble
Donald Summers, Los Alamos National Laboratory
Llewellyn Williams, Environmental Monitoring Systems Laboratory
10:30 am - Total Quality Management: What Does it Mean to the
Analytical Laboratory?
Jackson Hicks, Waste Treatment and Environmental Services
Department, Tennessee Eastman
11:15 am - Data Quality Assessment Overview
James Stemmle, Quality Assurance Management Staff
11:25 am - Data Quality Assessment: QAMS' Vision
Fred Haeberer, Quality Assurance Management Staff
Daniel Michael, Research Triangle Institute
1:00 pm - Assessment of Data Usability in Superfund
-------
Larry Reed, Hazardous Site Evaluation Division, Office of Emergency
and Remedial Response
1:30 pm - Assessing Data Quality for Agency Pesticide Decisions
Richard Schmitt, Health Effects Division, Office of Pesticide
Programs
1:55 pm - Assessment of Error in Soil Data
Jeff van Ee, Environmental Monitoring Systems Laboratory, Las Vegas
2:45 pm - Using Power Analysis to Assess Office of Water Data
Quality
Robert Graebner, TetraTech
3:10 pm - Facilitated Discussion Groups
4:15 pm - Discussion Group Summaries
4:30 pm - QAMS Wrap Up
James Stemmle, Quality Assurance Management Staff
4:40 pm - Adjourn
6:30 pm - Awards Banquet
- Ralph Bauer, Deputy Regional Administrator, Region 5
- Presentation, of the Quality Assurance Manager of the Year
Award
Wednesday, February 12
8:30 am - Training Sessions
1:30 pm - Training Sessions repeated
Thursday, February 13
8:30 am - Common Interest Group Sessions
Friday, February 14
8:30 am - Common Interest Group Reports, Closing
11
-------
MEETING HIGHLIGHTS
Nancy Wentworth, Director, EPA Quality Assurance Management Staff,
announced the theme of the meeting as "Customer Supplier
Understanding: Measuring Environmental Quality Successes." She
explained that speakers would be addressing the customer/supplier
relationship: how it is defined and how it can be measured. Ms.
Wentworth also gave a brief update on current activities and future
plans for the Quality Assurance Management Staff.
Eugene Rouleau, Deputy Regional Director, U.S. Office of Personnel
Management, discussed the importance of good customer/supplier
relationships. He emphasized that every contact with the customer
shapes their view of the supplier, and that customers look at their
relationships with suppliers, not just what they provide.
Fred Haeberer, a Quality Assurance Management Staff member, gave a
brief presentation on the Environmental Monitoring Management
Courcil (EMMC): how it was formed, its purpose and organizational
structure, and its progress to date. In a later presentation with
Daniel Michael of Research Triangle Institute, Dr. Haeberer
discussed the actions that QAMS is taking to address the issues and
challenges of the data quality assessment process.
Gary Johnson/ a Quality Assurance Management Staff member, reported
on the current status of ANSI/PSQC E4 and discussed the purpose and
implementation of the document.
Steve Hufford, Chief, Information Management Branch, Office of
Information Resources Management, explained the six data-related
standards, pointed out the challenges in the data standardization
process, and discussed future directions in data standardization.
Jackson Hicks, Superintendent of Waste Treatment and Environmental
Services, Eastman Kodak, Tennessee Division, described the basic
principles and methodology for effectively implementing TQM in an
organization.
Daniel Michael, Research Triangle Institute, defined data quality
assessment, examined the three forms of input in the data quality
assessment process, and discussed several issues and challenges
currently facing the data quality assessment process.
Larry Reed, EPA Hazardous Site Evaluation Division (HSED), Office
of Emergency and Remedial Response, explained HSED's role in the
assessment of data usability of Superfund. He discussed two new
guidances: "Data Usability in Risk Assessment," and "Data
Usability in Site Assessment" and emphasized their importance to
Superfund.
iii
-------
Richard Schmitt, EPA Health Effects Division, Office of Pesticide
Programs, described the Office of Pesticide Program's role in
reviewing data, the types of data that are reviewed, and the QA-
related activities that are utilized to assure data of the right
type and quality.
Jeff van Ee, EPA Environmental Monitoring Systems Laboratory in Las
Vegas, discussed bias and variability in soil sampling: the types,
ways of identifying, assessment, and methods of reducing.
Robert Graebner, Senior Scientist, Tetra Tech, discussed the use of
statistical power analysis in assessing data quality, and its role
in data quality management.
PANEL DISCUSSION:
Dr. Lindsey Wood, Total Quality Manager, Regulatory and Clinical
Development, Proctor & Gamble; Donald summers, Los Alamos National
Laboratory; and Dr. Llewelyn Williams/ Senior Science Advisor, EPA
Environmental Monitoring Systems Laboratory, Las Vegas participated
in a panel discussion on "Merging Quality into Science and
Research." Each panelist gave his own perspective and experiences
of the benefits of implementing a quality assurance program into
fields of science and research.
IV
-------
SPEAKER BIOGRAPHIES
-------
Ralph (Dick) Bauer is the Deputy Regional Administrator of Region
5 for the Environmental Protection Agency, where he assists the
Regional Administrator in carrying out air, water, hazardous
waste, and other pollution control programs in Illinois, Indiana,
Michigan, Minnesota, Ohio, and Wisconsin. With a B.A. in
Bacteriology, Mr. Bauer presently serves on EPA task forces on
budget reform and multicultural diversity, and is chair of the
EPA Quality Improvement Board.
Linne Bourget is the president of Positive Management
Communications Systems, a company specializing in positive
approaches to organizational change, such as vision-based
planning, interpersonal communication, group dynamics, and
leadership development. Dr. Bourget has a Ph.D. in Social
Psychology/Organizational Change, and has 25 years of experience
in management and organizational consulting and training.
Jeff van Ee is employed at the EPA Environmental Monitoring
Systems Laboratory in Las Vegas, where he works on field sampling
methods and quality assurance for soil studies. He has been with
EPA for over 20 years, during which time he has developed quality
assurance and monitoring systems for the Los Angeles Reactive
Pollutant Program and the Regional Air Pollution Study; led EPA
in the development of solar-powered monitoring equipment and
satellite-based telemetry in remote areas; and developed
monitoring approaches for locating abandoned oil and gas wells,
geophysical and soil-gas techniques for hazardous waste site
assessments, and monitored underground storage tanks.
Robert Graebner is a senior scientist for Tetra Tech. He is
currently working on several database applications, including a
dredged material tracking system (DMATS) for EPA Region 9, a data
management system for the baseline ecological assessment of
disposal activities for NOAA, an entry system for the EPA Ocean
Data Evaluation System (ODES), and a project tracking system for
the National Estuary Program (NEPTUNE). Mr. Graebner has an M.S.
in Wildland Resource Science/Biometrics and is a member of the
American Statistical Association and the Association of
Environmental Professionals.
Fred Haeberer is a chemist with EPA's Quality Assurance
Management Staff. He has 11 years of experience in analytical
methods development at EPA and the U.S. Department of
Agriculture, and is currently involved in the Environmental
Monitoring Management Council (EMMC). Dr. Haeberer has an M.S.
in Organic Chemistry and a Ph.D. in Physical Organic Chemistry.
-------
Jackson Hicks is the Superintendent of the Waste Treatment and
Environmental Services Department at Eastman Kodak, Tennessee
Division. He has worked as an analytical chemist for the West
Virginia Pulp and Paper Company and the Eastman Chemical Company,
and has over 27 years of experience in analytical chemistry,
waste treatment, and department management. Dr.. Hicks has an
M.S. in Physical Organic Chemistry and a Ph.D. in Analytical
Chemistry.
Steve Hufford is the Chief of the Information Management Branch
in EPA's Office of Information Resources Management (OIRM). He
is responsible for agency-wide programs in form management,
information security, data administration/standards, strategic
IRM planning, IRM policy and reviews, Privacy Act compliance, and
maintaining EPA's application systems inventory. Mr. Hufford has
B.A. in Biology and an M.S. in Ecology from Duke University.
Gary Johnson is a member of the Quality Assurance Management
Staff, where he develops and makes improvements on guality
management policies and procedures. He co-authored ANSI/ASQC-E4-
19xx, the ASQC Energy and Environmental Quality Division effort
to develop a national consensus QA standard for environmental
programs, and has presented numerous papers on QA and quality
management in national and international symposia. Mr. Johnson
is currently the Associate Regional Councilor and Chairman of the
Environmental Restoration Committee for the ASQC Energy and
Environmental Quality Division.
Joanne Jorz is the vice-president of Program Development at
Conceptual Systems, Inc., a human resources consulting firm.
Prior to joining CSI, Ms. Jorz was Director of Human Resources
Development at JWK International Corporation, where she worked on
decentralizing personnel functions, designing non-classroom
training experiences and organizational analysis studies. She is
a member of NSPI, FETA, and ASTD, and was a recipient of the ASTD
TORCH award.
Daniel Michael manages Research Triangle Institute's Washington,
D.C.-based Environmental Research Planning Department, where he
facilitates the data quality objective (DQO) process for planning
and designing soil, ground-water, and surface water surveys at
Superfund sites. He has developed and co-presented seminars,
papers, and workshops related to DQO concepts, the most recent of
which was "Quantitative Decision-Making in Superfund, a Data
Quality Objectives Case Study," published in Hazardous Materials
Control. Mr. Michael is developing a generic set of procedures
to assess the sufficiency of data for use in decision-making.
VI
-------
Robert O'Brien is a senior statistician at the Statistical Policy
Branch of the Office of Policy, Planning, and Evaluation, where
he is currently developing statistical computing methods and
statistical hypothesis tests to evaluate chemical data at waste
sites. Mr. O'Brien has an M.S. in Mathematics, and worked for
the Bureau of Census for nine years in survey design and analysis
before joining EPA in 1986.
Mary Ann Pierce is the project manager for environmental programs
at JWK International Corporation, where she has provided training
support to EPA's Quality Assurance program for the past six
years. Ms. Pierce has 15 years of experience in instructional
design and training, and developed various print, video, and
classroom instructional materials, including the Train-the-
Trainer workshops for EPA.
Larry Reed is the Director of the Hazardous Site Evaluation
Division in the Office of Emergency and Remedial Response. He
has been an EPA employee since 1974, when he worked for the
Office of Planning and Evaluation, the Office Pesticide and Toxic
Substances, and the Office of Water Permits and Enforcement. Mr.
Reed has an M.P.A. from the JFK School of Government, Harvard
University, and an A.B. from Youngstown State University.
Eugene Rouleau is the Deputy Regional Director for the Office of
Personnel Management for the Dallas Region. Mr. Rouleau has an
M.S. degree in Personnel from George Washington University, and
has been a government employee for nearly 30 years. He worked
for Senator Mac Mathias and Congressman Bill Steiger as a Fellow
of the American Political Science Association in 1987, and was an
employee of the U.S. Marketing Group of Xerox through the
President's Executive Exchange Program.
Richard Schmitt is the Acting Deputy Division Director of the EPA
Office of Pesticide Programs, Health Effects Division. He has
worked in the Office of Pesticide Programs for 19 years, and has
a Ph.D. in Chemistry from the University of California at
Riverside.
James Stemmle is an environmental scientist with the Quality
Assurance Management Staff, where he manages the annual national
meeting on managing environmental data quality, reviews QA
Program Plans, and promotes Data Quality Objectives. Dr. Stemmle
has a Ph.D. in Physical Inorganic Chemistry from Catholic
University, and has been with EPA since 1974.
Donald summers is the leader of the Quality Operations Office at
the Los Alamos National Laboratory, where he is currently
establishing a lab-wide quality program. Mr. Summers has 29
vii
-------
years of experience in quality assurance and has developed and
managed quality programs for nuclear and non-nuclear activities.
He is the Secretary for the Subcommittee on Waste Management for
ASME/NQA-1 and the Chairman of the Work Group for Low Level
Waste.
John Warren is a senior statistician and Acting Chief of the
Statistical Policy Branch of the Office of Policy, Planning, and
Evaluation. With a Ph.D. in Statistics, Dr. Warren is currently
developing efficient sampling schemes for the investigation of
hazardous waste sites. His current efforts were presented at the
agency's Eighth Conference on Statistics in March.
Nancy Wentworth is the Director of the Quality Assurance
Management Staff at the Environmental Protection Agency. She has
worked for EPA since 1978, and has received the EPA Bronze Medal
for Commendable Service (1991). Ms. Wentworth has a B.S. degree
in Civil Engineering from Tufts University and an M.S. in
Engineering from Pennsylvania State University.
Llewellyn Williams is the Senior Science Advisor for EPA's
Environmental Monitoring Systems Laboratory in Las Vegas, where
he works on improving monitoring and measurement methods and data
quality assurance. With a Ph.D. in Zoology, Dr. Williams taught
at Rutgers and Fordham Universities before joining EPA in 1972 as
a senior limnologist for the National Eutrophication Survey. He
has authored and co-authored over 100 reports and scientific
publications.
Lindsey Wood is the Total Quality Manager in Regulatory and
Clinical Development at Proctor and Gamble. He has a Ph.D. in
Microbiology, and was an assistant professor in infectious
diseases at the University of Texas Medical School in Houston.
Dr. Wood has published over 30 papers and abstracts on infectious
diseases and is a member of the Infectious Diseases Society of
America.
Vlll
-------
INTRODUCTION
-------
INTRODUCTORY SPEAKERS
NANCY WENTWORTH, DIRECTOR OF THE QUALITY ASSURANCE
MANAGEMENT STAFF/ began the meeting at 1 p.m. on Monday
afternoon. She announced the theme as "Customer Supplier
Understanding: Measuring Environmental Quality Successes," and
stated that speakers would be addressing how customer supplier
relations can be defined, and how they can be measured. She
further described the role of QAMS in the presentations as
"trying to help you, our customer community, to better understand
what we're doing, how the agency is operating these days, and the
directions of the EPA Quality Assurance program."
Nancy then introduced LOUIS SAAVEDRA, MAYOR OF ALBUQUERQUE.
Mr. Saavedra welcomed the attendees to the city of Albuquergue,
and shared his concern on the topic of data quality and TQM. He
showed his interest in preserving the environment, and emphasized
that we must take a "dispassionate, rational, and scientific look
at the issues." The speech ended with this thought: "When all
is said and done, you have an incredible balancing act to
undergo; and because the results of your work will perhaps not be
revealed until you and I are long gone, our children and
grandchildren are the ones for whom we are saving this land."
-------
PRESENTATIONS
-------
-------
KEYNOTE ADDRESS
EUGENE ROULEAU
DEPUTY REGIONAL DIRECTOR
U.S. OFFICE OF PERSONNEL MANAGEMENT
"CUSTOMER-SUPPLIER RELATIONSHIPS"
First, I want to ask you a few questions: How many of you
engage in some form of physical exercise each week? How many are
into running? OK, how many run longer than an hour at a time as
your longest run during the week? I run marathons. People who
are into exercise, but aren't into distance running, wonder what
we talk about when we're out there. This is what happened
several weeks ago in Dallas with a friend of mine, Leo.
Leo is a roofing contractor. He works through a broker that
has an agreement with a national chain of retail stores. Leo
said to me, "Gene, I had such a tough week; I'm thinking about
canning the business, uprooting the family, and moving to some
other place." Now, Leo is usually a very upbeat, energetic, and
optimistic person, so I asked him what was going on. He said,
"I've got business management problems. I'm getting whipsawed.
The broker's got my beeper number and he calls all day long.
I've got three different levels at the retail store chain across
the city where I get my business and I've got sales people, the
managers on home improvement departments, and their total
division manager calling me. They're asking all sorts of
questions and constantly giving me negative feedback. I just
don't feel good about these relationships. And on the other
side, I talk to my workers on the job sites, and they're trying
to get the work done, and what are they looking for? I haven't
done this for them, I haven't done that for them...What can I do
about this?"
We talked about the situation, and came up with some
solutions:
1. Reduce the time he spends on his mobile telephone — not
be so accessible to so many people all of the time. Instead, buy
mobile radios to talk to his crews, and only give his beeper
number to a few people. He rented a voice mail box and opened up
communications with retail customers. Now, customers can leave a
message on Leo's voice mailbox, and he can pick up the messages
every 15 to 20 minutes. Before he calls back, he contacts the
job site to find out what's going on (has the customer said
anything?). He asks the workers about the status of the work in
progress, so that when he calls the customer back, he can tell
them that either the problem has been solved, or that it is in
the process of being solved. Leo also tells customers that they
can talk to his crew leader at any time on job to ensure that
their expectations are being met.
2. Empower and incent his worker to pay more attention to
what the customer wants. If a worker hears from a retail
-------
customer that there is a problem, and that worker takes actions
to solve it, Leo will reward him with a bonus.
3. Meet with every retail customer before his crew begins
to talk about what they have and have not bought according to the
contract.
4. Listen to customers' special expectations for him and
his crew. Leo gets rid of expectations that cannot be satisfied,
gets everything out in the open from the beginning, and confirms
when the job is to be finished.
5. Authorize the workers to spend up to $50 (in time and
materials) beyond what the contract calls for to fix things that
are unrelated, but that may be a problem for the customer.
After one month of following these guidelines, Leo reduced
call-backs at customer sites by 80%. He now spends about 65% of
his time with the customers, discussing their needs and
expectations. In short, Leo is not behind a desk or driving
around in his truck all day; he is negotiating and understanding
customer expectations.
Several lessons can be picked up from Leo's experience. In
the words of Ben Franklin: "We are what we repeatedly do.
Excellence, then, is not an act; it's a habit." If you get into
an excellent habit of relating yourself and your resources to
what your customer needs, then suddenly, having a quality service
or product is not difficult; it's a natural outgrowth of the way
that we become. Vince Lombardi: "Practice doesn't make perfect,
perfect practice makes perfect." If you begin to think about
that customer (if you are the supplier), and if you think in
terms of satisfying that customer's requirements, everything we
do when we meet with the customer either leads toward the goal or
away from the goal. You either get closer to the customer, or
further away.
Dr. Michael Leboeuf, the author of How to Win Customers and
Keep Them for Life; "Any organization, any industry's most
valuable asset is its stock of satisfied customers." How
satisfied are the customers of the data quality industry? Is the
industry improving? Is your business improving based on customer
satisfaction? Dr. Leboeuf says that the greatest business secret
in the world is that rewarded customers buy, multiply, and come
back. We reward customers every time we have contact with them
in a way that's so human and yet so competent, that they look
forward to doing business specifically with us.
One study from Texas A&M looked at 900 customers of
diversified businesses and found out that what customers want
from service delivery outfits is reliability, most of all.
Customers expect us to be timely, responsive to their needs,
accessible, available, credible, able to explain our stock and
trade (from technical jargon to regular English), and willing to
help them when they have problems. They expect quality; they
expect it to be attractive to do business with us; and they
expect to be treated as special individuals. Customers weigh how
our people treat them in every part of the relationship.
-------
Dr. Leboeuf observes that "Customer's perceptions are
customer's realities." Their perceptions of how we serve them is
the difference between what the customer gets from us versus what
they expected to get from us. If .customers perceive that the
service we offer is reliable, on time, of high quality, and
responsive, we're going to win them and keep them with us for a
long business relationship.
When we look at the data quality industry in each of our
business units within that industry, we need to see our industry
in our own businesses through the eyes of the customers. Every
contact that we have with customers shapes their views of us and
it either increases their interest in doing business with us at
the same volume and value, or decreases their business with us.
Those are the consequences of every transaction between us and
our customers.
The customers are valuable to us as people in the industry,
because the customers have the insights and the understanding to
tell us how we're doing and how we can get better. Customers
aren't shy once you ask them what they think. If you don't have
a well thought-out plan or process for collecting and analyzing
customer satisfaction data, think about this: What is your
method of assessing customer satisfaction? How about taking the
information you collect from customers and translating it into
the next phase (in the TQM sense) of customer requirements?
Then, validate the requirements by asking the customers if these
requirements are real for them. Begin the process by measuring
how well your business is doing, and how well the industry is
doing in meeting customer expectations and requirements.
What are some of the tools you can consider if you want to
get customer satisfaction data? Develop short survey
questionnaires for employees and customers that ask what their
ideas are; and standardize short interview lists to be used
during telephone conversations and in-person meetings with
customers. The process of asking customers what they think about
our services tells them that we value their views.
After listening to customers and improving our existing
products and services, the next step is to work on identifying
the customer's unmet needs. How do you get started in that
direction? Ask customers and potential customers what services
and products they would like to have, but either can't get, or
can't get at a price they can afford. Based on what they tell
us, we first try to stretch one of our existing products or
services to meet their needs. We try to retool and repackage,
change formats, etc. If that doesn't work, we brainstorm new
products and services.
The relationship between customer and supplier is part of
something bigger — what is called "the service revolution." One
of the proverbs in the Bible says it best: "Without a vision,
people perish." The thinker with the most influence on the
vision of the service revolution is Karl Albrecht, co-author of
Service America and author of At America's Service. Each of us
knows that seven out of ten Americans earn money in the service
-------
sector of our economy. We also know that the data quality
industry is part of the service sector of the American economy.
But does the industry, and each organization within the industry,
have a clear vision of managing the services that we provide to
customers? Albrecht says that the quality of service
increasingly makes the difference in which organizations are
effective in the market place and in the public sector. He and
Dr. Leboeuf agree that the customer's perceptions of the quality
of service can be managed. It's not a random event; it's
something that we can manage and participate in.
Our customers look at the relationships with our people, not
just what we provide. We can manage myopically by thinking that
if we have a quality product or service, it will sell itself.
This is no longer true in the service sector of the economy.
It's also tough to understand that controlling the quality of a
service or product is not like controlling the quality of a
physical product. Why? Because the person who provides that
service is such an integral part of the service delivery, and
it's the human element that is so difficult to control.
What do we do if we want to get with it in the service
sector of the economy? What is the unifying idea behind what we
do? Our strategy must be to focus our attention on the real
priorities and needs of the customer, both current and unmet.
Next, we need to have a customer-oriented front line of people
who help customers get what they paid for, and create good will
for our organizations. We must act on problems that come up in
the systems that managers created; clean up systems so they
decontrol people who are directly involved with customers; and
make our work process system (that backs up service delivery
people) more user-friendly. Any time a customer says they had
trouble getting in to your people about an accounts receivable, a
warranty, etc., it's not good.
One last observation from Albrecht: In the service
business, we need to think in terms of employees having
transactions with customers instead of simply operating in
predefined roles and stereotypes that say "customer
representative #1 will do the following things..." That isn't
the way we want to create jobs for our people who face the
customers. Our employees are engaged with customers in running
transactions; they are not in predefined jobs where everything is
cut and dried.
-------
FRED HAEBERER
QUALITY ASSURANCE MANAGEMENT STAFF
U.S. ENVIRONMENTAL PROTECTION AGENCY
"THE ENVIRONMENTAL MONITORING MANAGEMENT COUNCIL"
Fred Haeberer reported on the Environmental Monitoring
Management Council (EMMC): its formation, purpose, organizational
structure, and progress to date.
The EMMC was created in 1990 to "coordinate the agency's
methods and research and development efforts to foster
consistency and simplicity in the methods that were being
developed and the methods that were already on board; to
coordinate short and long term methods and development strategies
to promote the adoption of new technology and new methodology; to
coordinate QA and QC guidelines as they apply to specific
methods; to evaluate the feasibility and advisability of a
national laboratory accreditation program; and to coordinate
other activities influencing the agency's environmental
monitoring needs."
The Council reports directly to the Deputy Administrator,
Hank Habicht. Below the DA is the Policy Council, the group
responsible for identifying and addressing monitoring issues and
needs, and formulating policies. It oversees the Steering
Committee, which, in turn, oversees the activities of the five Ad
Hoc panels.
When the EMMC was created in 1990, the Steering Committee
was asked to address five key issues. Ad Hoc panels were then
appointed to focus on these issues:
1. The agency's QA Services group — How are these
materials supplied and how are we going to fund
them?
2. A Methods Integration group — assigned to
look at the methods that utilized very similar
methodologies for analyzing the same analyte, but
were slightly different for different programs or
different matrices.
3. The Methods Compendium group — responsible for
identifying and developing software which would
provide information on the agency's methods — What
methods are available, what methods are in
progress, what analytes are affected, etc.
4. The Quality Assurance in Regulation Development
group — tasked with identifying and developing a
process for ensuring that the agency's regulations,
as they are developed and promulgated,
will be backed by environmental monitoring methods
that will support the regulatory requirements.
5. The National Laboratory Accreditation group —
responsible for examining the feasibility and
advisability of a national environmental laboratory
accreditation program.
-------
The five figures below present the co-chairs, the charter,
and the accomplishments and future steps of each Ad Hoc panel.
EMMC
Quality Assurance Services
Co-Chain Thomas Hadd
Office of Research Profram Management
William Hum
Office of Air Quality Planning and Standards
Chamr Address the Issue of Sustaining Adequate
Funding for QA Semces and ResearcB by ORD.
AuuiiyliihmsnB QA Semces A Research Needs Survey.
Draft Repon, and Recommendations. £^
Men Sops Fpal Repon and Recommendations for -oaMar^,
FY 1993 Budget Proposal (Spring 1991 >• V^^V
EMMC
Metods
Co-Chain
Charter
Accompbsnmenu
Next Steps
UnyReed
Office of Emergency and Remedial Response
On vuu. Environmental Services Division
Region m
Evaluate the Feasibility of Standardizing
Analytical Methods across Media and Programs.
Methods integration Process
Three .Method Imefraucfl Pilot Projects Cnderwiy
Method Integration Work Croup Established
Five Integrated Methods Complete
(September 1991).
EMMC
Automated Methods Compendium
Cc-Chain
Charter
Next Steps
BiUTeUianl
Office of Wa»r
Quality Assurance Management Staff
InsnutionaUz* the List of Lists as an Agency- Wide
Tracking System for Analytical Methods.
Read-Only Version of EMM Distributed.
Region m Pilot Test
Complete Update. Analyie Survey Result*, aflai
and Add Method Precision and Bias Data.f*sMBlpl*4
EMMC
Analytical Methods & Regulation Development
CoThaifi Magae Thieian
OflS of Regional Operations sod Saue/Uxal Relations
Carol Wood. Emiiesanaaul Semces Division
Rettoal
Chan* Develop a Sysom for Factoring Methods De»tluuimiu
•^ValMewiiCananis ia>o OM Agency s Refulauon
OVVMOpBMtf PWCWaV
AcoorapuabnaaBB Praccis co M unpHDiotta ^^^.
3rd QuanerFY 1991. a^%
^^^/
1
EMMC
National Laboratory Accreditation
Co-Chan Jim Fmfer. Eimrermemal Semces Division
RefionlV
Ramona Trovato
Office of Water
Oaner Explore the Feasibility of a Uniform. National
Accreditation Proeraoi for Laboratories
Perfornuni Envmnmental Tesunf Procedure!.
Accomplishments Phase I Study of Benefits and Potential
Issues Complete
Not Steps Establisn FACA Corramnee (April 1991
Phase U Study of User Needs and
Options (Summer 199J).
8
-------
The final five minutes of the presentation were devoted to
the current status of the Environmental Monitoring Methods Index
(EMMI), a cross-referencing tool that contains information on
agency lists, analytes, and analytical methods. A pilot test was
done with Region III in September to allow experts to review this
software and suggest changes, additions, etc. Based on the
feedback, Version 1 of EMMI was produced and selectively
distributed in January.
EPA employees can obtain a copy of EMMI by contacting EPA's
ALL-IN-1 electronic mail system at 1-800-334-2405. Employees
from other Federal agencies should contact Bill Steltz at (202)
260-7120. EMMI is not yet available to the private sector, but
privatizing efforts are being made so it can be distributed and
sold to the general public.
-------
10
-------
GARY JOHNSON
QUALITY ASSURANCE MANAGEMENT STAFF
U.S. ENVIRONMENTAL PROTECTION AGENCY
"ANSI/ASQC E4, THE QUALITY ASSURANCE REQUIREMENTS FOR QA SYSTEMS"
In this presentation, Gary Johnson reported on the current status
of proposed ANSI/ASQC E4, and discussed the stages of developing
the document, from conception to implementation.
He began the discussion by stating that as a national
standard for QA systems, ANSI/ASQC E4 strives to bring "harmony
and consistency" to environmental decisions in the Federal
Government, and to eliminate the dilemma of "...ten regions
having ten different ways of looking at the same problem."
Gary explained that the absence of an existing set of
criteria for government agencies induced the creation of a
special group under the auspices of the American Society for
Quality Control (ASQC). Their goal was to develop a national
consensus standard for environmental programs, a process that
could be used by all agencies within the Federal Government.
During the search for a new approach, the group looked at "... a
lot of good pieces of things that different groups had applied to
environmental programs." They decided that in order to come up
with the best QA standard possible, they would "...look at the
best everyone had to offer, and steal shamelessly from all the
good ideas that we could find."
Starting from scratch, the group took a TQM approach and
began with the basic principles. They organized a scheme that
focused on three parts: Planning, Implementation, and
Assessment. Based on this design, "The standard will tell what
needs to be done. It's not going to say how or by whom; that's
your part. You have to develop a QA program to meet the needs of
your organization."
Gary concluded with the current status of ANSI/ASQC E4. The
first draft was released in March, 1991 at the International
Waste Management Conference in Las Vegas. It was then
distributed at the EPA QA meeting in Dallas. Senior level
individuals from DOD, DOE, EPA, the contractor community, and the
nuclear regulatory committee commented on the initial draft. By
September, the draft was ready to undergo the wide review process
through ASQC. More than six hundred specific organizations and
individuals received a ballot, where they were asked to review
the standard, make comments, and mark it acceptable or
unacceptable. The closure date was January 31, but because of
the enormous amount of review requests, a second ballot may occur
in May or June. The standard is expected to be approved and
published later this summer.
11
-------
12
-------
STEVE HUFFORD
CHIEF
INFORMATION MANAGEMENT BRANCH
OFFICE OF INFORMATION RESOURCES MANAGEMENT
U.S. ENVIRONMENTAL PROTECTION AGENCY
"AGENCY-WIDE DATA STANDARDS"
Steve Hufford began his presentation by briefly explaining
the six basic data related standards. The first mentioned was
the Chemical Abstract Service (CAS), used by EMMI as a registry
numbers standard for identifying compounds. The second was a
standard coding scheme to identify regulated facilities across
the agency. The idea, according to Mr. Hufford, was "that
establishing a common coding scheme for the regulated facilities
could help EPA share data across the program boundaries." The
third was "the locational data that deals with the use of
latitude and longitude coordinates to identify where things are
in the environment and also deals with noting down the method
used to obtain the coordinate, the description of what the
coordinates present, and an estimate of the accuracy of the
coordinate." The fourth standard dealt with the essential data
elements to take whenever a ground water sample is collected.
Mr. Hufford added that "this minimum set of data elements is
important to ensure good use of the parametric data..." The last
two focused on standards for transmitting data rather than
particular values of the data. The fifth dealt with transmitting
measurement data from laboratories; and the sixth was a recently
developed policy on electronic reporting. "This policy says, to
the extent possible, use the ANSI X12 family of standards or
transaction sets to automate the process, or if appropriate, use
EDIFAC, a more internationally used set of standards."
Mr. Hufford then pointed out the challenges in the data
standardization process. He emphasized that "what we're trying
to do in the data standardization program is, very thoughtfully
and with a lot of outreach and education, strike the appropriate
balance between rigidity and flexibility..." He continued on to
say that enforcing standards is an ongoing challenge, and the key
to success is education and awareness. "I am particularly
pleased about the formal amendment to the EPAAR, EPA's
acquisitions regulations. The amendment requires compliance with
IRM policies and standards, not only the data standards, but also
other IRM-related guidance and policies, such as guidance for
systems design and development." Mr. Hufford further noted one
last challenge: that the process for creating standards is slow.
"The process requiring absolute consensus tends to create
standards that are either vague or generic or difficult to
interpret. We're grappling with those and trying to make
implementation and guidance as clear as we can."
The final portion of the presentation was devoted to future
directions in data standardization. Mr. Hufford explained that
the basic idea is to link data standards into a more broad
13
-------
context of data administration. "A lot of parts of EPA deal with
organisms, in one way or another, and a standardized way of
communicating that information might be worthwhile." He further
assured the audience that "a number of efforts are already
underway."
In closing, Mr. Hufford mentioned that his branch, in the
Office of Information Resources Management, currently maintains a
computerized inventory of EPA's applications system. The
inventory includes a summary and descriptive information on more
than five hundred applications. While not highly detailed, the
system "does serve as a useful pointer."
14
-------
NANCY WENTWORTH
DIRECTOR
QUALITY ASSURANCE MANAGEMENT STAFF
U.S. ENVIRONMENTAL PROTECTION AGENCY
"THE NEWS FROM QAMS"
"The goal of this meeting is for you all to know what we in
QAMS are doing — what our current activities are, and what the
plans are for the near and long-term future."
Ms. Wentworth began the presentation with her vision of
quality assurance in the agency as "an excellent management
program for environmental data collection through planning,
implementation, and assessment activities which produce data of
the appropriate type and quality for EPA decisions and for EPA
decision-makers and research managers through efficient and cost-
effective processes." She added that the main considerations of
QA must focus on timing, politics, budgets, and perceptions of
those outside the agency.
Ms. Wentworth continued on to describe the ways in which
QAMS is approaching this vision. First, she explained that
senior management knowledge and attention to quality issues
across the agency are being improved. To this effect, she and
the Deputy Assistant Administrator for the Office of Research and
Development (ORD), John Skinner are meeting regularly to focus on
these issues. Although John could not attend the meeting, he
wanted Ms. Wentworth to relay several points to the audience:
1. "ORD recognizes and takes very seriously its
responsibility for QA across the agency."
2. ORD has declared QA documentation to be a material
weakness under the Federal Manager's Financial Integrity
Act (FMFIA).
3. QA is an integral part of the internal management control
process, a system that requires quarterly reports to the
ORD Senior Management Council, which Skinner chairs.
4. QAMS should be given credit for "being in the vanguard of
TQM implementation in the agency, and by showing that TQM
can work in a technical decision-making process."
5. We must have decisions based on data and fact. "We can
no longer make our decisions based on anything but
strong, scientifically defensible, efficiently-generated
scientific information.
Ms. Wentworth then gave a brief report on the panel that was
organized one year ago to look at the role of science in the
agency. Although the report from the panel was not yet available
for release, several recommendations were cited that the
administrator has accepted — "recommendations dealing with how
we do science, how we plan, how we manage, and how we coordinate
internally." According to Ms. Wentworth, a particularly relevant
proposal to the quality assurance community was "the specific
recognition that the culture of the agency needed to change so
15
-------
that it valued science." She further added that "...science is
not a corollary to our job; it is primary to our job." Another
recommendation was that there needs to be QA in peer reviews for
all scientific and technical efforts across the Agency, including
the regional offices. The report and an implementation plan to
respond to these recommendations were released to Office
Directors in March.
A brief discourse on the manager's role in attaining the QA
vision was then given. Ms. Wentworth emphasized that managers
provide an example of quality by knowing the QA program
requirements; providing guidance/assistance on QA issues to co-
workers and the public; identifying emerging issues/problems that
may be Agency-wide; and assisting QAMS in developing new QA tools
for application across the Agency.
Ms. Wentworth then gave a summary of specific QAMS
activities. She added to Fred Haeberer's speech on the EMMC by
saying "...through the methods integration panel's quality
control sub-committee, we're trying to standardize terminology
and definitions for things like detection limits, quantitation
limits, reporting limits, and that strange and mystical term,
validation of a method. I bet there are 175 different
definitions for validation. We are trying to bring some
standardization and common implementation of that understanding
across the Agency." She added that Gary Johnson was currently
working on revisions to the management systems review guidance
and the management/program plan documentation, and that efforts
were being made to write a new data quality objectives guidance
document for Henry Longest, the Director in the Superfund office.
"We are also doing case studies in regions across the country to
be able to append real examples of data quality objectives for
soil cleanup, sediment, groundwater, etc. to this guidance
document." Ms. Wentworth went on to describe several projects
with the Department of Energy: data quality objectives, case
studies, a compendium of radiological and mixed waste methods,
and alternate designs for quality control programs for attaining
different levels of defined data quality. She then mentioned a
cooperative agreement with the National Academy of Science,
Mathematical Sciences Division. "NAS is doing a program review
of our quality assurance program and looking at whether the
approaches we are taking in statistical design are successful in
the context of teaching them an experimental design." Going back
to the QA documentation issue, Ms. Wentworth commented that the
first step in the development and oversight of the QA corrective
action process is to make sure that the documentation is in place
across the agency; the second step is to assess the quality of
the programs.
Ms. Wentworth ended her presentation by reiterating the need
to work together, and the importance of senior management
involvement. She also emphasized the significance of success
stories: "...this is one way to get positive attention, and
improve your perception and standing in the Agency."
16
-------
JACKSON HICKS
SUPERINTENDENT
WASTE TREATMENT AND ENVIRONMENTAL SERVICES
EASTMAN KODAK - TENNESSEE DIVISION
"TOTAL QUALITY MANAGEMENT: WHAT DOES IT MEAN TO THE ANALYTICAL
LABORATORY?"
"Let's assume that you want to institute Total Quality
Management (TQM) in your organization. My first question to
you is why?"
Jackson Hicks explained that there is only one reason to
embrace TQM: to improve vour business. If you implement TQM for
other reasons, such as to win the Deming or Malcolm Baldrige
awards, to impress customers with the "avant garde" position of
yo'ir business, or to use in advertisements, stated Dr. Hicks,
"you are destined for disappointment. It will not sustain the
process and will lead to unsatisfactory results."
How, then, do you begin? "I can't tell you exactly what to
do; there is no secret step-by-step formula. If you follow some
general guidelines, you can implement TQM effectively in your
organization." Dr. Hicks continued on to give four basic
principles for implementing TQM:
1. Never lose sight of the fact (and your focus) that you
are improving your business and satisfying your
customers.
2. Keep it simple and involve the total workforce.
3. Use all of the quality management tools that are
appropriate to help accomplish what you want.
4. Be evolutionary rather than revolutionary.
He explained that the implementation process must begin at
the top and filter down throughout the organization. Successful
TQM cannot happen without involving all of the employees; and all
employees can't be involved without total commitment from the top
manager all the way down. "If the parts are to join and build a
true framework, then the entire organization must embrace the
concept."
Figure 1 on the following page represents a system of
interlocking teams, the process that Dr. Hicks attributes as the
reason for the success of the Waste Treatment and Environmental
Services Department.
17
-------
Technical
Supervision
Team
Figure 1
Interlocking Teams
• Business Focused Team
• Manage The Day To Day Business
. Maintain And Develop Process Skills
• Develop And Use Interpersonal Skill
(Citizenship)
• Expand Boundaries As Team Matures
• Develop Business Focus
• Long-Term/Short-Term Strategic Planning
• Provide Technical Leadership To Meet Ever
Changing Regulations And Requirements
• External And Internal Results Assessment
• Planning And Managing Past Today
• Transfer Technology
« Remove Barriers
• Provides Vision And Business Philosophy And Focus
• Manages By Principles
• Provides Support And Resources For All Teams
• Long-Term Strategic Planning, Establishes Goals
And Direction
• Personnel Development And Planning
Figure 2 shows the method of implementing TQM, and outlines
the major stepT-^in the process for each team.
Management Team
Education
»
Analysis
|
r- Planning
t
Execution
»
Check
•
L- Act
r
•/
Coil
FIGURE 2
STRATEGY FOR INVOLVING EMPLOYEES
IN MANAGING/IMPROVING THE BUSINESS
.......
BU1CM
•«
••
Education
Analysis
*
Planning
»
Execution
Check
*
Act
•*• —
'cLui*
•«
r
Education
*
Analysis
Planning
Execution
*
Check
Act
The first step, EDUCATION, involves reading materials on
successful TQM processes (Deming, Juran, etc.)/ watching videos
that relay success stories of businesses, and visiting other
companies that are using TQM. "Such visits help you see what
others are doing, aid in developing support networks, and
establish benchmarking. You will want to learn from the best,
and shouldn't be afraid to steal shamelessly."
The second step is ANALYSIS. This stage involves conducting
a business focus evaluation where the management team reviews the
entire business of the organization. Management then determines
what modifications are needed and focuses on what teams are
needed and what boundaries should be established for these teams.
18
-------
PLANNING is the third step. In this stage, the management
team develops a list of business improvement plans and promotes a
communication plan to share the concept at all levels. The plan
contains provisions to providing training, education, and any
additional resources that may be required.
The management team begins the Plan-Do-Check-Act cycle, and
the plan is rolled down to the next level team until every
employee has been involved. "This is not a quick process and may
require several years to evolve. The entire process is
evolutionary and is always changing as continual improvements are
made."
While Dr. Hicks admitted that he could talk about the
implementation process for "quite a while," he changed the topic
back to what TQM means to the laboratory. In his own words, the
TQM system represents:
o Commitment v. Compliance
o Make It Work v. Did My Part
o Best Ever v. Standard
o Continual Improvement v. Status Quo
o Team Work v. Individual
o Cooperation v. Competition
o Celebration v. Chewing Out
o A Way Of Life v. A Program
o Every Employee As A Manager v. Management Control
"In TQM organizations, employees have input into decisions
that affect them. They are given responsibilities and they
accept accountability for their actions. They are not part of
the organization; they are the organization."
Dr. Hicks stated that the real rewards and incentives to the
employee was being a part of the process. "Ham biscuits,
baseball caps, pins...they serve a purpose. But the real reward
comes when a team feels real satisfaction and realizes, 'Hey,
we've improved our process; we're part of the system!"1
Dr. Hicks continued to describe some accomplishments that
have occurred since using the TQM system. "The result of team
efforts is that during our last inspection involving our
incinerator, our wastewater treatment operations, our analytical
laboratory's support, the federal and state inspectors could not
find a single item that was not in compliance. This had never
happened before. Our analytical laboratory is responsible for
collecting and analyzing all of the samples required to show
environmental compliance. This amounts to over 11,000 points
each year. During 1988, we lost 143 pieces of data; in 1989, we
lost 148. These data points had to be reported to state and
federal regulatory agencies as exceptions and were subject to
regulatory fines. A team of our analysts and samplers then began
a project to eliminate exceptions. They met their goal of 0
exceptions in 1990. Since the project began in January 1990,
there has only been one exception, which occurred in March of
1991. To date, we have lost one piece of data in twenty-five
months — one piece out of 22,900."
19
-------
An additional accomplishment, Dr. Hicks added, was the
facility and investigation plan that Tennessee Eastman Division
was required to develop in early 1990. "The services analytical
lab initiated a plan to develop and implement a quality assurance
project plan. As a unit team, consisting of laboratory
supervisors, chemists, analysts, and sample collectors, they were
asked to study the requirements and guidelines for establishing a
project plan, and determine how the plan could most effectively
be implemented. The team developed and documented a complete
plan that would meet the requirements of written procedures,
QA/QC checks, chain of custody procedures and forms,
identification of personnel, etc. The plan was completed in less
than six months and has been successfully integrated into the
total laboratory program."
Dr. Hicks' last words emphasized the importance of Total
Quality Management: "I believe that TQM will be sustainable
because it is not a fixed system, but rather a system of
continual improvement. It involves the best of many systems and
it develops an atmosphere where individuals can take pride in
their work and ga^n personal satisfaction from performing their
jobs. It's a team approach where the organization's and the
employee's needs can all be addressed. It involves the best of
quality control and the integration of technology and sociology."
"Do I think TQM is for everybody? The answer is yes."
20
-------
PANEL DISCUSSION
"Merging Quality into Science
-------
MODERATOR:
GARY JOHNSON
QUALITY ASSURANCE MANAGEMENT STAFF
U.S. ENVIRONMENTAL PROTECTION AGENCY
PANELISTS:
DR. LINDSEY WOOD
TOTAL QUALITY MANAGER
REGULATORY AND CLINICAL DEVELOPMENT
PROCTOR & GAMBLE
DONALD SUMMERS
LOS ALAMOS NATIONAL LABORATORY
DR. LLEWELYN WILLIAMS
SENIOR SCIENCE ADVISOR
ENVIRONMENTAL MONITORING SYSTEMS LABORATORY
U.S. ENVIRONMENTAL PROTECTION AGENCY
21
-------
OPENING REMARKS
GARY JOHNSON:
"Each of these gentlemen I will introduce will share their
perspectives on the activity of merging quality into science and
research. Our first speaker will be Dr. Lindsey Wood from
Proctor & Gamble in Cincinnati. Lindsey will share his
perspective on integrating quality and total quality management
into the aspects of the health care division of Proctor &
Gamble's research and development work. Following Lindsey will
be Don Summers from the Los Alamos Laboratory, a 29-year veteran
of quality assurance. Dr. Llew Williams, the senior science
advisor at EPA's Environmental Monitoring Systems Laboratory in
Las Vegas, will be our last panelist. Llew will present a
perspective on merging quality in science from EPA's viewpoint.
Each panelist will give a 10-minute presentation. Following
that, I will pose a series of questions for them to address. And
lastly, we will open the floor for questions from the audience."
DR. LINDSEY WOOD:
Dr. Wood targeted the five key activities that must be in
place to have a successful total quality program:
1. Strategy/Results Focus. "You have to combine the ideas of
quality initiatives or research projects with improving systems
or improving capabilities; and focus improvement efforts on the
capabilities most critical to business results. For example, our
management team made a list of processes that needed to be
improved, and we geared our training around those suggestions."
2. Leadership. "The Director must be a leading advocate and a
role model. He has to practice total quality, or the whole thing
falls apart. Also, process improvements need to be part of the
project reviews, and it is critical to get mid-level managers on
board. And finally, you've got to get out there with concrete
successes and actively market those to your organization."
3. Training. "Training, of course, is critical. We've used the
Plan Do Check Act (PDCA) cycle, which is actually the scientific
method restated. We use trained facilitators, because it's not
good having amateurs teaching amateurs. And finally, you have to
document progress and learnings that each group comes up with.
Their current best approach must be circulated to the other
groups, so different divisions aren't reinventing the wheel."
4. Innovation Teams. "The innovation teams must have the right
people involved and the right team size. We recommend teams of
six to ten people, so it's small enough to keep working, but not
too big to get confusing. These teams must be empowered to make
changes; management has to be prepared to let them run the
experiment. And finally, successes have to be celebrated and
rewarded. For instance, we have a project team of the year, and
a process improvement team of the year. These are things where a
team may get a dinner certificate for $100."
22
-------
5. Measurement. "The teams have to be charged with clear
objectives and result measures. We tally these up with monthly
and quarterly progress reviews. In fact, participation in these
teams and the results they get have become part of our written
'what counts' factors."
Dr. Wood summarized his presentation with some clear-cut
results of using the success model. "What we've seen in health
and personal care is a much sharper focus on the customer; better
technology transfer; reduced cycle time; reduced costs; and
improved employee satisfaction."
DON SUMMERS:
Mr. Summers used several personal experiences to relay the
benefits of a quality assurance program. The first occurred in a
facility at Los Alamos, where he studied a metal pressure vessel.
"I didn't know what it was, but as a certified welder, I knew
that there should be a weld on the neck of the vessel. I
couldn't see it, but I knew it was there." He learned that the
welder haii not been certified and thus, the procedure was not
qualified. "It defied QA principles. You have to have
traceability to the materials, and you have to have a procedure
qualification record. But there it was, and it was working." A
quality assurance program in that particular situation, Summers
told the audience, would have provided the necessary
documentation for others to come along and repeat the process.
It would, in essence, have allowed "the information to be passed
on, other than by word of mouth, to the people who were going to
come behind them."
Another more recent incident took place while Mr. Summers
was meeting with individuals who performed criticality
experiments. One of the group leaders asked him why they needed
quality. In response to Summer's request, the leader explained
that they built devices that detected radiation, each of which
cost millions of dollars. She further added that "they had no
design control, but they knew what they were doing." After
delving a little deeper, Summers learned that they did, indeed,
have design control — in the form of peer review meetings.
Summers concluded his presentation by emphasizing that
scientists and the community have to look at what they're doing
and recognize the merits of a formal quality program. He added
that you can't push a quality program, but if you start working
with people and leading the process, it will work.
LLEWELYN WILLIAMS:
Llew Williams opened his presentation by citing several
misconceptions about quality assurance:
1. That it assures quality, "...quality assurance procedures
don't assure quality; people assure quality."
2. That quality can be legislated.
3. That elevating quality is all positive. "If you elevate
23
-------
quality assurance to a great independent operation, you take the
responsibility away from the individual at the bench. He is the
person responsible for quality."
4. That quality is more a millstone than a milestone. "They
don't feel that the imposition of quality assurance procedures on
their research will improve what they are doing. They only feel
that it will be decreative and slow them down."
The topic of discussion then turned to the barriers to
quality and science. The first barrier, according to Williams,
is the lack of management commitment. "This may be a very
personal lack of commitment, even in the face of an
organizational statement." Lack of vision and visualization was
also cited as a problem. "Vince Lombardi once said that if you
can't visualize the result of a play, of exactly how it should
look if it's successful, you will never be able to coach it
successfully." A third barrier mentioned was a poor
understanding of what quality is. "Documentation is great to
show what we've done, but if we can stress the fact that by
bringing quality to our scientists, we can bring them an enhanced
level of satisfaction in their work, we ray be able to hook
them." Finally, the disintegration of personal value systems
such as pride in workmanship, job satisfaction, and client
satisfaction, was named. "If there are low quality expectations,
then people are willing to accept less. A problem in maintaining
high personal values in light of some of the institutional values
(those that stress profit at the expense of quality, and an
obsession with liability), is that they have a quality assurance
program for all the wrong reasons."
Llew also included several recommendations for changing
these values: quality-oriented training and education;
inspirational leadership; creating a low-risk environment to
bring out creativity; rewarding quality ethics; and synchronizing
personal institutional visions.
The presentation ended with a poem:
The operative word is quality,
And oh, what a ness we are in,
'Cause everyone want in, they all look at us,
And we don't quite know where to begin.
You see, quality's not just the seal on the lid,
Or the slop in the pocket of slacks;
And there's no guarantee that you'll get what you want
At a K-Mart, a Sears, or a Saks.
But ask them, "Exactly what is it you want
In a product, a service, or such?"
Any they'll likely respond with the strangest of looks
And insist that "You're asking too much!"
"Just dial in the settings and turn the thing on,
Then there won't be a human to err."
So we dial in the settings, and we get the garbage out,
Cause there wasn't a human to care.
24
-------
QUESTIONS
Quality is frequently seen bv scientists and researchers as not
being applicable to them. What ways or approaches have you found
to be successful in raising the awareness and appreciation of
quality among these groups?
LINDSEY:
"At the start of training, we have everyone do a time log for a
week to determine how much time they spend doing the right thing
right. It turned out that many of the groups came out at 60 to
70%. Then we turn that around and say, 'wouldn't you like to
have 40% more time to do research and not do these things that
aren't of value'? Scientists turn on to quality when they
realize that it's going to give them more time to do creative
science. We focus an^that and rell it as a benefit."
DON:
"We use the same approach that Lindsey mentioned. We emphasize
that a good quality program will allow them more time to do their
science and research, but it will also help them reproduce what
they've done."
LLEW:
"Many researchers have a difficult time dealing with data quality
objectives and their application to a tree-like structure where
there are a lot of decisions in research and no fluid operation.
I ask them to partition their work, look at each critical
decision point and what kind of information they need, and decide
what level of quality they are going to demand. Once they've
divided it into pieces, suddenly the data quality objective
concept works."
What do you see as some of the specific obstacles or barriers
that hinder the acceptance of quality among scientists and
researchers? Do you think these barriers are real or are they
merely perceived?
LINDSEY:
"One obstacle is the familiar 'I already do good work.' Another
is that some scientists feel that quality implies a loss of
control. To answer the second part of the question, I would say
that because barriers are perceived, they are real."
25
-------
DON:
"The biggest obstacle in promoting quality is that we have a
quality control inspector with two years of experience telling
the superintendent with thirty years of experience how to place
concrete or structural steel. We drove up the cost tremendously.
Quality costs too much money — at least that's the perception.
We have to promote quality as making money, or at least saving
money. And it can be done."
LLEW:
"The major barrier is getting scientists to understand what
quality can do for them. One area to explore is the concept of
quality assurance teams, where the objective is to leave the
scientist improved in the way that he doesn't do things lust to
satisfy your bookkeeping as a quality assurance professional."
Much of what we see and hear with respect to total quality
management places great emphasis on measuring success. Is it
practical to emphasize performance measurement in R&D. given the
inherent uncertainties involved. If so. how can research
performance be measured in a meaningful way?
LINDSEY:
"What we need to do in R&D is break things down into bite-sized
pieces. Look at what you can measure to determine if your
processes are getting more successful: turn-around time,
milestones to meet, percentages, etc. It's not good enough to
say that it's impossible to measure research. It is, but it's
not easy."
DON:
"In an R&D environment, total quality management and quality
assurance will provide the ability to reproduce what has been
done. So many times in an experiment, the final result pops out,
but we're not sure how it happened. So we have to go back and
duplicate it. But once again, R&D is very difficult to measure."
LLEW:
"I think that if we're going to measure and monitor, we've got to
take a giant step back and ask why we're measuring and
monitoring. We have to find a way to feed back those measurement
monitoring processes in a way that is clear to the scientist that
this is done on behalf of and in support of his science."
26
-------
Traditional R&D depends greatly on peer review as the means of
determining the quality or acceptability of a research product.
In what ways can quality be more effectively integrated with the
peer review process?
LINDSEY:
"We don't have any real problem with peer review fitting into
total quality; they're integrated. As part of the plan cycle,
it's automatic."
DON:
"If you get quality established when you start determining the
design phases, and start the review process at the same time, the
project will be successful."
LLEW:
** «*.
"Peer review should start at the conception and design of an
experiment or study. Bring in someone from the outside to
determine if you're doing things right. But, bring him in
early."
27
-------
28
-------
DATA QUALITY ASSESSMENT
-------
"DATA QUALITY ASSESSMENT"
Tuesday afternoon's presentations were devoted to the topic
of Data Quality Assessment (DQA). The chair, James Stemmle from
the Quality Assurance Management Staff, explained that QAMS is
heavily involved in DQA, and is currently developing guidance for
agency-wide use. Having recognized that assessment of data
quality is "going on everywhere," several important areas were
targeted for discussion. The first area was presented by Dan
Michael of Research Triangle Institute (RTI) and Fred Haeberer of
QAMS. Mr. Michael focused on RTI's support to QAMS' current
activities and ongoing research, while Dr. Haeberer discussed
QAMS' vision. Following that was "Assessment of Data Usability
in Superfund" from Larry Reed, Office of Emergency & Remedial
Response; "Assessing Data Quality for Agency Pesticide Decisions"
from Richard Schmitt, Office of Pesticide Programs; "Assessment
of Error in Soil Data" from Jeff van Ee, Environmental Monitoring
Systems Laboratory, Las Vegas; and "Using Power Analysis to
Assess Office of Water Data Quality" from Robert Graebner of *. ^.
TetraTech.
Later in the afternoon, following the presentations, six
facilitated discussion groups convened to discuss and formulate
answers to five questions. Each group then designated a speaker
to address their response to the question.
29
-------
30
-------
DANIEL MICHAEL
RESEARCH TRIANGLE INSTITUTE
Dan Michael began his presentation by asking, "Why worry
about data quality?" He answered this question, by giving several
reasons: 1. To preserve the integrity needed in the product
we're delivering; 2. To ensure customer satisfaction (spend time
developing measures of performance that reflect the customer's
needs); 3. To maintain credibility (failure to generate data
which allows us to take action decreases our believability); and
4. To make acceptably accurate decisions.
Expanding upon these reasons, Mr. Michael stated that "data
of inadequate quality will lead to higher probabilities that we
fail to take action when we should or we take actions
inappropriately when our budgets and our taxpayer's precious
dollars could be better spent somewhere else. If we fail to take
action when we should, we are exposing the public to unacceptable
risks. So, the consequences of making incorrect decisions cause
us to focus on data quality."
Mr. Michael then defined data quality assessment (DQA) as "a
process for evaluating a data set to determine whether the data
are appropriate for supporting a specific decision." He
emphasized that "supporting a specific decision" provides the
relative criteria to establish what quality means. "The key is
the strength of the decision and our ability to compare data or a
data set to some performance standards so we can determine if
those data are appropriate." DQAs that include the level of
uncertainty that is acceptable to the data user, he added, are
important to the whole process of data quality assessment.
The audience learned that the data quality assessment
process contains three forms of input: Data Quality Objectives
(DQOs), Data, and Statistics. As planning inputs, all three
represent the "DO" or "EXECUTE" phase of the PLAN-DO-CHECK-ACT
cycle. Mr. Michael examined each input to determine its
composition:
1. Data Quality Objectives: "The outputs of the DQO process
include a statement of a problem or decision to be addressed, the
input variables, the characteristics we need to measure, and the
criteria against which we are going to compare them to the
boundaries of where and when we are going to sample. The
decision rule addresses how we are going to use data." The
answer to this question, Mr. Michael says, is in how we are going
to use the data to make a decision and in determining how much
uncertainty can be tolerated in that decision. "Did we collect
data that would describe/define the correct variables? Do we
have the variable that we need? Were they collected in the right
location to represent the scale from which we want to make the
decision? According to Mr. Michael, these are all important
criteria against which the final data set should be measured.
2. Data: "We have to think about how data are generated and
about the error that can creep into the data set throughout the
31
-------
data collection process. The questions are: What amount of
error exists in one of those numbers? How does that affect the
overall total study error which is going to affect the degree of
certainty that we have in making a decision?" Mr. Michael
emphasized that if total error is too great because not enough
samples were taken or there was too much error in the subsampling
or analytical process, then the next question must be: Where is
the major source of error in this particular study and what do we
do about it?
3. Statistics: "We need to think about the assumption that the
data corrected for bias can serve as a zero bias data set, which
can then be used in a statistical test. Perfection in
environmental studies is finding out how the substance that we're
looking at actually varies in nature." Mr. Michael conceded that
studies will never be completely error-free, even when using
extensive sampling and the best technology for analyzing data.
He added that "what we need to know is just how variable the data
are and what effect that variance has on making the right
decision."
Mr. Michael then discussed key issues and challenges for
achieving QAMS' vision of data quality assessment. "We look at
it in two domains: the technical side and the management side.
From the technical side, we know what information and data are
required to perform the data quality assessment. It's not a very
straight-forward answer, because it depends on how you define the
goals of that data quality assessment, and if you can diagnose
any problems that may be contributing to the problem or
insufficiencies. On the management side, we need to develop
management procedures and communication tools to ensure success.
Obviously, getting to the bottom line can be a somewhat
threatening process. If you had never known that there was more
uncertainty in data than you would have been satisfied with, then
why should you want to know it now? It's through the process of
analysis and education that we hope to overcome these potential
snags."
Another key issue involves statistics. According to Mr.
Michael, QAMS' statistical vision can be achieved by formulating
the right hypothesis (making sure the decision is based on the
answer to the right question — is it stated succinctly enough to
determine whether there are adequate data to prove or disprove
the hypothesis?); determining acceptable error rates;
establishing the best statistic to be used (ones that will be
appropriate, given valid assumptions); and evaluating whether the
statistical design assumptions are valid.
A third issue/challenge is total study error. Several
recommendations include estimating total study error directly by
looking at the generated data set and noting its distribution and
variance; identifying the components of total study error (what
must be in place in the QC program to ensure enough data to
estimate the major sources of error?); determining when estimates
of the error components are needed and building that into the
design of the programs so the QC data will be readily available;
32
-------
and evaluating which components contributed the most to total
study error.
The last issue discussed came under the heading of
management:
1. Communicating with the statistician (when and how do we
ask for this type of assessment?)
2. Interpreting and using the results (how are we going to
ensure that they give us the answers to the questions
that most affect us?)
3. Defining the roles and responsibilities among
participants
4. Recognizing and overcoming the threats and obstacles to
DQA.
33
-------
34
-------
FRED HAEBERER
QUALITY ASSURANCE MANAGEMENT STAFF
U.S. ENVIRONMENTAL PROTECTION AGENCY
Following Dan Michael's discussion on the issues and
challenges impacting data quality assessment, Fred Haeberer
presented the actions that QAMS is taking to address those issues
and challenges. He emphasized four main areas of action:
Retrospective DQA, Total Study Error Analysis, Improving QC
Design, and Future Work.
Under the first area, Retrospective DQA, Dr. Haeberer
defined the development of statistical procedures and the
cultivation of a user-friendly guide for project managers as the
near term actions. A report on statistical procedures has been
drafted by the Research Triangle Institute (RTI), and plans are
being pursued for an updated version, with input and comments
from various reviewers. The user-friendly DQA guide has not yet
gotten off the ground, but it is on the current agenda. Long-
tenrfQ/K^^include performing retrospective analyses based on
case studies using existing data sets. "Our idea is to
demonstrate the methodologies that we propose in the guides in
the statistical procedures and to assure ourselves and you that
they do, indeed, work."
Beneath the heading of Total Study Error Analysis, Dr.
Haeberer explained that RTI completed a working diagram of error
partitioning last year. "We've got the estimation procedure and
equations documented at this point, and our long-term goals are
to go into existing data sets to perform case studies, and to
assure ourselves that these partitioning procedures actually
work."
Under Improving QC Design, Dr. Haeberer stated that QAMS is
"...trying to link the QC of sampling and laboratory analyses to
what is actually required by the data user. The intent is to
design the QC operations so that the resulting data will support
the decisions by achieving the required error levels. Our long-
term goal is to develop QC design procedures that can be used
with the DQO process to directly address management's concerns
and to come up with designs that address, and are responsive to,
the data quality criteria and data quality constraints that
management requires."
Dr. Haeberer stated that Future Work would include
documenting the lessons learned from analyzing case studies. He
then encouraged individuals with relevant case studies or data
sets to join in a collaborative effort of QAMS working with its
customers.
35
-------
36
-------
LARRY REED
HAZARDOUS SITE EVALUATION DIVISION
OFFICE OF EMERGENCY AND REMEDIAL RESPONSE
U.S. ENVIRONMENTAL PROTECTION AGENCY
"ASSESSMENT OF DATA USABILITY IN SUPERFUND"
Larry Reed works as the director of one of the four
divisions at the Superfund Headquarters office. His division,
the Hazardous Site Evaluation Division (HSED), runs the front end
of Superfund, "providing policies, guidances, and services to the
regions, which implement Superfund clean-ups." In a brief
description of Superfund, Mr. Reed stated that "Superfund is not
a delegated program. Unlike many of the other environmental
programs that have a delegation requirement or process built into
the statute, we don't have such a thing. We're trying to build a
state partnership within the limits of the statute, but much of
what we do in Superfund, we either have to do with the regions or
contractor staff."
The Superfund process begins with Preliminary
Assessment/Site Investigation (PA/SI), where the site is ranked
using the Hazardous Ranking System (HRS) to determine if the site
should be on the National Priority List (NPL). Relative
questions of this phase of the process are: Is there a release?
What is it? What are the pathways of exposure? After the site
is placed on the NPL, the Remedial Investigation/Feasibility
Study (RI/FS) process begins under the direction of the Hazardous
Site Control Division (HSCD), which ultimately leads to a Record
of Decision (ROD) regarding the disposition of the site. HSED
provides analytical services by the Analytical Operations Branch
(AOB) through the Contract Laboratory Program (CLP), while risk
assessment guidance and services are supplied by the Toxics
Integration Branch. Relevant questions include: What are the
contaminants of concern? What are their concentrations? What
are their boundaries? Is there danger to the public health or
the environment? Where should we place monitoring wells?
Additional questions for the baseline risk assessment during the
RI/FS are: What are the exposure pathways? What is the exposed
population? What is the estimate of magnitude, duration, and
frequency of exposure for each receptor group? Are there
contaminants with toxic effects? Are they at levels that are a
problem? Record of Decisions are based on answers to: On what
analytes should we focus remediation? What level is our clean-up
goal? What interferes with our proposed remedies? The third
phase, the Remedial Design/Remedial Action, begins when a ROD has
been developed. During this construction and clean-up period,
analytical services are provided by HSED. Questions of concern
are: Is it clean? Is it still clean?
37
-------
"In addition to providing contractor services for analyses
of samples, and methods development to keep up with Superfund
needs and state-of-the-art technology, HSED is also responsible
for providing guidance for data assessment."
HSED first provides Contract Compliance Screening for all
CLP data. This process uses a computer to check contract
deliverables against the requirements, and determines appropriate
payment for services. HSED also develops, maintains, and updates
"Functional Guidelines for Review of CLP Data," a document that
establishes a standard way to evaluate and review data and
qualify it for use.
Another guidance, RI/FS DQO, was produced in 1986 to take
advantage of the data quality objective (DQO) process by: l.
determining what to do with data that had not been collected
using DQOs, since there was no predetermined level of uncertainty
for sampling or analysis; and 2. identifying data quality needs
for major Superfund uses.
In 1988, HSED initiated the Data Usability Workgroup, which
produced the draft guidance for "Data Usability in Risk
Assessment." Designed to "provide data users with a nationally
consistent basis for making decisions about the minimum quality
and quantity of environmental analytical data that are sufficient
to support Superfund decisions," this guidance helps identify
performance measures and sets acceptable limits for confidence
level, power, and minimum detectable relative difference. It
also provides strategies for designing sampling plans and
describes how to select an appropriate sample design strategy for
these questions: What is present and how much? Is it different
from the background? Are all pathways identified and examined?
Are all pathways fully characterized? "Data Usability in Risk
Assessment" has completed peer review and is currently being
finalized.
An additional guidance, "Data Usability in Site Assessment,"
was sent out for review in October of 1991. It covers planning
and assessment issues related to the generation and use of
analytical data, and provides "generic data use categories with
specification of analytical data quality parameters required for
that use category." Relevant questions in this guidance are
related to ground water releases to aquifers, surface water
releases to surface waterbodies, soil exposure contamination of
surface materials, and air releases as gases or particulates to
air.
Mr. Reed stated that "Our message to you is that HSED is
dedicated to identifying appropriate data needs for Superfund and
providing analytical services to meet these needs, including
analyses, planning guidances, and assistance/guidance in
assessment of the data."
Mr. Reed concluded his presentation by emphasizing that the
Hazardous Site Evaluation Division would like to know the
audience's perception of the Data Usability Guidance: What would
they like to see in the guidance? How can the HSED help identify
and get the right data for their uses?
38
-------
RICHARD SCHMITT
HEALTH EFFECTS DIVISION
OFFICE OF PESTICIDE PROGRAMS
U.S. ENVIRONMENTAL PROTECTION AGENCY
"ASSESSING DATA QUALITY FOR AGENCY PESTICIDE DECISIONS"
Mr. Schmitt started his presentation by explaining the
function of the EPA's Pesticide Programs. He stated that
although these programs do not generate much data, they do review
large volumes of data. The reason for this, he added, was that
the Office of Pesticide Programs registers pesticides
(registration is basically a license to sell a pesticide in the
United States). "Before a pesticide can be sold or distributed
in this country, it must be registered by EPA. And before this
can happen, we require that chemical companies such as Dupont
develop enough data to allow us to decide if the pesticide is
safe."
The discussion then turned to the types of data that the
Office of Pesticide Programs (OPP) reviews: Product Chemistry -
reviews data on the composition of pesticides, including
concentrations of impurities such as dioxin; Residue Chemistry -
reviews data on the chemistry of the pesticide related to
residues in food such as beef; Product Performance - evaluates
information that is required to make sure that a product works
(e.g. kills the weed as it should); and Plant Performance - data
to show that the pesticide will not harm the crop it is trying to
protect.
In terms of data volume, Mr. Schmitt explained that OPP
receives roughly 12,000 studies a year, ranging from a few pages
on the physical properties of a pesticide to a four-foot stack of
data on a long-term chronic feeding study for toxicology
purposes. "There are about 650 pesticide chemicals currently
registered, thousands of inert ingredients [chemicals that are
added to pesticide formulations, but are not active by
themselves], and about 24,000 pesticide products registered by
the EPA."
The final section of the presentation was devoted to the
question, "How do we assure that the data we get are of the right
type and quality?" Mr. Schmitt explained that they utilize nine
QA-related activities:
1. Regulations/Guidelines. "Title 40 of the Code of
Federal Regulations (CFR), Part 158, contains a list of studies
that the Office of Pesticide Programs requires for registering a
pesticide. It lists the types of studies, and gives information
on when the studies are required. Different types of data are
required for different types of usage. For instance, a pesticide
that's used to protect the seed while it's in the ground has
different data requirements than a pesticide that is supplied by
an airplane to a growing crop. The location or use of the
pesticide is also an important factor in determining what data
39
-------
requirements are necessary. An indoor pesticide, such as a
cockroach spray, for example, has entirely different data
requirements than a pesticide applied to soy beans, where
contamination of the environment is more likely. We require an
even larger data set for food pesticides, including large residue
data and toxicology data requirements."
2. Data Reporting Guidelines. "The data reporting
guidelines are an addendum to each set of guidelines, and are
used as instructions to the data generators on how to format
their studies. The purpose of these guidelines is to make sure
that all the data are included in the study, and are easy for the
scientists to find."
3. Standard Evaluation Procedures. "The standard
evaluation procedures are a set of guidelines that instruct our
OPP scientists how to review the studies. We developed these
procedures so that two scientists reviewing the same study would
come up with the same conclusion."
4. Phase III Guidance Package. "The Phase III Guidance
package is a recent set of criteria designed to instruct the
people who develop data on how to decide if EPA will reject the
study. These were developed for what we call re-registration."
5. Re-Registration Conferences. "Since data requirements
were less stringent 20 or 30 years ago, Congress has mandated
that EPA bring the databases for these chemicals up to standard.
Chemical companies cannot sell their pesticides and make a profit
until EPA registers the pesticide. That, in itself, is a strong
incentive to develop good data. For products already registered
and currently being sold, there is little incentive to develop
good data to keep them on the market, because EPA is not very
good at taking pesticides off the market. So, we developed what
is called acceptance criteria. Registrants must use these
criteria to decide if their data are acceptable. If they submit
a study they think is acceptable, and we decide it's not, then we
can take their product off the market."
6. Screening Procedures. "...a series of screening tests
that registrants must pass before we accept their data. These
consist of administrative screens and science screens.
Administrative screens ensure that the submissions are complete,
legible, and contain all the confidential business information
and good laboratory practice forms that we require.
Surprisingly, about 80% of submissions for new chemicals fail
this administrative screen. Science screens make sure that all
the required studies are there, and data are suitable for
review."
7. Good Laboratory Practices (GLP). "We require GLP
certification for each study. We cooperate with our Office of
Compliance Monitoring in carrying out laboratory and study
audits. If one of our scientists sees a study that looks
suspicious, we will recommend to the Office of Compliance
Monitoring that the study be audited to check for legitimacy."
40
-------
8. Study Rejection. "If the study is not up to our
standards, we will not register the pesticide. In some cases,
information is left out on how the study is conducted; in other
cases, they may have to go back to the laboratory and generate
additional data or analyze more samples. Or, in extreme cases, a
whole study may have to be repeated."
41
-------
42
-------
JEFF VAN EE
ENVIRONMENTAL MONITORING SYSTEMS LABORATORY, LAS VEGAS
U.S. ENVIRONMENTAL PROTECTION AGENCY
"ASSESSMENT OF ERROR IN SOIL DATA"
Jeff van Ee focused his presentation on the subject of bias
and variability in soil sampling: types of, ways of identifying,
assessment of (using QA/QC), and methods of reducing.
He began with measurement quality objectives, a relatively
new term similar to data quality objectives (DQOs). The chief
difference between the two, however, is that DQOs "apply to the
bigger part of the problem," (i.e. to the total acceptable
error — design, sampling, and measurement) while measurement
quality objectives focus on a smaller sampling site. "A data
quality objective might be posed to you that someone wants to
know the average concentration at a particular site with a
certain degree of confidence and accuracy. But, depending upon
where you go at that site and how you sample, you could get any
number you want." With measurement quality objectives, new
terminology has been introduced to narrow the sampling field.
One term, exposure unit, refers to pinpointing a certain area of
concern. "Depending upon how the contaminant is distributed
across the site, you can get any number. So, during the process,
we bring in statisticians, soil scientists, and the people making
the measurements in the lab, and try to narrow the process down
to: what area of the site do you want to know the average
concentration and to what degree of confidence?" Another term is
called remediation unit. "The concept is if there is
contamination, how small of an area does it have to be in before
you will do something about it? With the heterogenous nature of
soil, you can have contamination in a very small volume,
contamination that may be missed by your sampling grid design, or
because of the particular tool you're using to collect samples.
We try to pin people down and say, 'OK, if we were to miss a hot
spot in our sampling design, what size would it have to be before
it's a cause for concern?"1
Mr. van Ee then mentioned a document that the Environmental
Monitoring Systems Laboratory (EMSL)-Las Vegas has recently
completed. As the lead author, he describes the purpose of the
document: "It defines the numerous sources of bias that can
occur in a soil sampling study. Since assessing bias in a soil
sampling study can change over time, from contractor to
contractor, and laboratory to laboratory, we're hoping to assess
the total measurement bias in sampling and analysis." He
emphasized that variability and systematic error (i.e. bias) can
be the result of specific choice of tools ("...contaminants are
typically spread across particle size distributions, and
depending upon the type of sieve you use, you could be rejecting
all of the contaminants"); handling, transportation, and
preparation of the sample; and the analytical laboratory.
43
-------
"By pairing samples—taking a routine sample and taking a
sample right next to it, which we call a field duplicate—and
analyzing those samples separately, we can begin to get a handle
on some of these sources of variability." According to Mr. van
Ee, these sources can be identified by using blank samples and
field evaluation samples. He added that the only way to control
variability is to determine the cause of that variability. This
can be partially achieved by a new computer program called
ASSESS. Modeled after a spreadsheet, ASSESS enables the user to
assess errors in the sampling of soils, and thereby identify
possible locations of bias and variability in the sampling
system.
To wrap up, Mr. van Ee mentioned several tools that can be
used to solve the bias and variability problem in soil sampling:
1. "Guide to Site and Soil Description for Hazardous Waste
Sites Characterization"—a document that contains over
900 definitions related to soils;
2. "Characterized Site Specific QA/QC Materials"—a new type
of QA/QC samples that consists of contaminated soil that
very closely _epresents the site being investigated; and
3. "From Risk Assessment to Remediation," a CD ROM that
holds all current guidance.
44
-------
ROBERT GRAEBNER
SENIOR SCIENTIST
TETRA TECH
"USING POWER ANALYSIS TO ASSESS OFFICE OF WATER DATA QUALITY"
Robert Graebner focused his presentation on the use of
statistical power analysis, the desired evaluation of
environmental monitoring programs, and the role of power analysis
in data quality management.
Mr. Graebner introduced the topic by stating that "sound
management decisions require data that are not only accurate but
sufficient." In other words, having data that are accurate is of
little use unless there is enough to produce a statistically
sound decision. He further expressed a strong belief in getting
managers and statisticians involved early in the planning process
so they can easily understand the system itself, the components
of the system, and how the components interact. This allows
managers to provide full knowledge on the type of decision to be
made - a decision vital to designing an effective environmental
monitoring system.
Mr. Graebner then described the five basic steps to
designing an environmental monitoring program as:
1. Developing program objectives. This also includes
defining objectives and making decisions that concur with
these objectives.
2. Establishing testable hypotheses. This involves
determining the questions that must be answered to support
sound management decisions, and then restructuring the
questions into testable hypotheses.
3. Selecting analytical methods. This includes selecting
analytical methods with the interaction of engineers,
scientists, and statisticians so that valid assumptions can
be made prior to sample design and collection.
4. Evaluating minimum detectable difference and power.
This step involves setting the criteria for data quality in
terms of an acceptable level of significance and power, and
an acceptable minimum detectable difference.
5. Selecting the number of sampling stations and replicates.
The remainder of the presentation focused on describing
basic statistical concepts to demonstrate the value and
importance of statistical power analysis in assessing and
managing environmental monitoring data. The following six pages
show how Mr. Graebner demonstrated the application of statistical
power analysis in assessing data quality.
45
-------
ENVIRONMENTAL MONITORING
PROGRAM DESIGN
DEVELOP PROGRAM
OBJECTIVES
i
ESTABLISH TESTABLE
HYPOTHESES
SELECT POTENTIAL
ANALYTICAL METHODS
I
EVALUATE MINIMUM
DETECTABLE DIFFERENCE
& POWER
SELECT NUMBER OF
SAMPLING STATIONS
& REPLICATES
ANOVA STATISTICAL MODEL
Wtwr»:
Observations at Station i ft. Replicate j of,
for example, the concantration
of • selected chemical
Maan of all Yw obsarvatlons
Effect of the I* laval of an environmental
factor (0.9... station location)
Random errors not accounted for by
either ti or T,
46
-------
HYPOTHESIS TESTING: POSSIBLE
CIRCUMSTANCES AND TEST OUTCOMES
O
C/D
O
LLJ
O
ACCEPT
REJECT
HYPOTHESIS
ACTUALLY TRUE ACTUALLY FALSE
STATISTICAL POWER ANALYSIS
ENVIRONMENTAL PROTECTION PERSPECTIVE
TYPE!ERROR (ft) Realdiffwvnctsexist
but «•
«.g. • A bioaccumulation monitoring
program may tail to ottoct •tevattd
concentrations of conuminants ta
fishttssu*.
STATBHCAL POWER (1-§) Probability of
— — mm^ »ih« «A^^
coiTvcny on
47
-------
PROBABILITY DENSITY OF THE F STATISTIC
-email
IB) MtMBMrO«l«yal'Whin MUl
POWER CALCULATIONS
,.X) d £
where:
v,
vx'
numerator degrees of freedom
denominator degrees of freedom
noncentrality parameter
48
-------
ODES STATISTICAL
POWER ANALYSIS
MARCH 1986
PREPARED FOR:
MARINE OPERATIONS DIVISION
OFFICE OF MARIf « AND ESTUARINE PTOTECTJON
US ENVIRONMENTAL PROTECTION AGENCY
WH-556M
WASHINGTON. DC 20460
PROBABILITY OF DETECTION vs.
SPECIFIED LEVEL OF DIFFERENCE
Pectinaria califomiensis
MINIMUM DETECTABLE DIFFERENCE (% OF MEAN)
49
-------
EFFECTS OF INCREASED SAMPLING EFFORT ON
THE POWER OF THE ONE-WAY ANOVA. NUMBER
OF REPLICATE SAMPLES AT EACH STATION:
A=10, B=5, C=3
1.0,
a 0.8 ^
O.J
0.4 J
0.2 J
0.0
(A)
0 40 10 120 160 200 240 280 320 360 400
MINIMUM DETECTABLE DIFFERENCE (% OF MEAN)
MINIMUM DETECTABLE DIFFERENCE
IN THE ABUNDANCE OF
Pectinaria californiensis
vs. NUMBER OF REPLICATES
NUMBEM OF REPUCATES
50
-------
Conceptualization Tools
Monitoring Guidance for the
National Estuary Program
51
-------
52
-------
AWARD BANQUET
-------
RALPH R. BAUER
DEPUTY REGIONAL ADMINISTRATOR
REGION 5
U.S. ENVIRONMENTAL PROTECTION AGENCY
"When Nancy asked whether I would come this evening and
address you, the answer had to be yes. I think the best way to
demonstrate my commitment to quality in the agency is the way I
spend my time.
I would like to address the question of who the customer is
for the Federal Government, but I'd also like to speak to some
broader issues: managing environmental data quality and its
relationship to customers and suppliers and measuring
environmental success. I have devoted a fair amount of effort
over the years to get the agency to focus on environmental
results rather than activities. If you will indulge me, I'd like
to talk a bit about that.
The whole context of my remarks will be couched in terms of
TQM. Since I am presently the Chairman of the agency's Quality
Improvement Board, I'm on a mission to talk about TQM and its
implementation in EPA. In that context, I will try to weave in
who the customer is.
Most of you know that EPA is the first agency in the Federal
Government that is committed to implementing TQM from top to
bottom in all of its organizations. Now, why is that? Why
should we do that? At. a meeting held about a year ago with the
Regional Administrators and Assistant Administrators, they
invited David Nadler to speak. Mr. Nadler is one of the
principals of Delta Corporation, an outfit in New York that
specializes in leading companies through big changes. When Mr.
Nadler was asked, what does it take to make TQM go in your
organization? he responded that the 'only place he had seen TQM
successfully implemented was in organizations that felt they had
no choice; that were going under if they didn't.'
That's certainly true of the private sector. What are the
hallmarks of quality in the private sector? Being a successful
company? The Fortune 500? One of the measures we use is whether
the companies are successfully competing in this country. If you
look at the list of companies that were in Fortune 500 from 1978
to 1988, 40% of those that were in Fortune 500 in 1978 were not
there in 1988. That's how rapidly things change. Another
indication of quality in the United States comes from Peters and
Waterman, the authors of In Search of Excellence. They profiled
44 companies that were considered to be the essence of quality.
Of those 44 companies, only 14 still meet the test of excellence,
according to the criteria of Peters and Waterman. Why is this
happening? Why are all these companies going out of business? I
think the answer is in a single word: CHANGE.
53
-------
What are some of the things that cause us to want to get
behind this quality move? The first is the fiscal crisis that
this country faces. Our industry is certainly hard pressed to
compete in an international market. So, what are the
implications of that to us? Rosabeth Kanter, the Editor of the
Harvard Business Review, uses the analogy of the croquet
tournament in Alice in Wonderland to describe the situation: the
mallets turn into flamingos, the wickets grow, the Queen is
yelling 'off with her head,' and they're constantly changing the
rules on people.
I think that's a reality of the business climate. If we
want to have a strong economy in this country, we at EPA are
going to have to be more responsive to our customers. Another
one of the fiscal crises is with our state agencies. Many of
them are in financial duress, and I think the problem will get
worse. I'm told that we may receive back some of the programs
that have been delegated to our states, because they simply can't
shoulder the load anymore. The states are another one of our
principle customers, if not the principle customer. And what do
they need from us? They need for us to eliminate a lot of the
inspections we do. If I were with the state, I think I would get
very tired of EPA continually inspecting everything we do.
We've got to look at these customer/supplier relationships
between us and the states and decide what can be eliminated.
Significant changes can be made. For one, we're going to have to
address the impending change in the workforce. There's going to
be a large increase in the entering workforce of women and
minorities. Statistics say that by the year 2000, white males
will represent only 15% of the new entrants in the workforce.
Also, young people today are not driven by the same workaholic
attitude about work that my generation suffered under. They're
looking for more balance in their lives, more of a say in how
they do work.
Another is the changing nature of pollution. When I first
got into this business, the pollution problems in this country
were a relatively small number of very large sources. In some
instances, you could smell the rivers before you got to them.
Well, thanks to the work of a lot of people in this room, we have
largely corrected those problems. We've made tremendous progress
in the last 20 or 30 years, and the remaining environmental
problems in this country are now a very large number of small
sources.
We're going to have to use different approaches to pollution
prevention and environmental education. There will always be a
place for command and control, but if we want to move the ball to
the next yard mark on the field, we're going to have to use a
different set of tools than we've been using. To me, that
implies a different approach. We're not going to get what we
want by commanding people to do it; we have to give them a good
reason to do it. As Kendall was attempting to do, we have to
create a vision, and get people to buy into that vision.
54
-------
Another change is the loss of confidence in the Government
as an institution. Just recently, we brought together all the
Assistant Administrators (AAs) and Regional Administrators (RAs)
to look at the question of how we were going to provide
leadership to the whole endeavor. We brought in a couple of
inspirational speakers, one of whom was Mark Roberts, an
economist from Harvard University. Dr. Roberts said that the
government has lost credibility. The public no longer feels that
the government is fair, and questions their effectiveness to
solve society's problems.
Dr. Roberts wrote a book stating that EPA asks all the wrong
questions. For instance, one of the wrong questions is, 'Is it
safe or is it not safe'? Although we tend to look at things as
either being safe or unsafe, the reality indicates that it's on a
continuum. You can't ever promise people things that are
absolutely safe. We can't give that kind of guarantee. You can
reduce the risk to a tolerable level, but you can't eliminate it
entirely.
On the question of fairness, Roberts uses the Johnson Space
Center in Houston. Does it really make any sense to the public
to have a space center in Houston? Is it only coincidental that
it was put there during the LBJ administration? I think most
people realize that there are politics that go into the decisions
that we make. And as a consequence, there is concern about the
fairness of our decisions. An environmental example is that
people are concerned that a disproportionate amount of risk in
this country is borne by the poor and disadvantaged. Black
people, for instance, have two to three times the amount of lead
in their blood levels than the non-black population. So,
environmental risks are clearly not fair across the population.
Effectiveness. Roberts says that this country isn't going
to come out of its economic doldrums in the near future. And in
the meantime, he added, real harm is being done to individuals.
The middle man is being stripped away, causing a significant loss
of standard of living for the middle income people. So, since
there is an increasing percentage of our population that
experiences this loss, is it any wonder that they are demanding
that their tax dollars be used for good purposes? I think that
we at EPA have to make sure that we are sensitive to this issue
when dealing with the regulated public, and responsive to their
concerns.
There are a lot of agencies out there competing for tax
dollars. And the polls show that the environment is definitely
not at the top of the list. Jobs, health care, education,
safety, etc. all rank above the environment; and as the economic
condition worsens, the environment continues to drop in
importance. Roberts says that the way out of the predicament is
to be credible, fair, and effective. I think TQM is the vehicle
to help us do that. We need to get in touch with our internal
and external customers to find out what they expect from us; and
we need to measure our successes in terms other than beans or
administrative actions.
55
-------
Another of my visions of success is that people will find
better and faster ways of doing business. We're going to have to
teach people new tricks: education, outreach, public/private
partnerships using economic incentives...! think these kinds of
things have great promise.
Lastly, I think success will occur when all our internal and
external processes involve fewer pointless hassles...when
companies can get permits from EPA in a timely way and with
enough information upfront so they're not guessing what is
necessary...when we have a helpful attitude about answering
questions... and when we can eliminate rework.
I'd like to finish my presentation with a true story that
has changed my thinking about the importance of data. About a
year ago, I went to an international conference on environmental
indicators in Florida. As a moderator for one of the sessions, I
decided to kill some time and sit in on a presentation by a
university professor. I can't remember his name, but he was
talking about a mathematical model of climate change in North
America. He had created and built the model, but decided that he
needed to establish its credibility and veracity. He did this by
running the model backward to determine if it would predict the
weather pattern in the past. He ran the model all the way back
to the last ice age, some 20,000 years ago, and found that
vegetation was extant during those years. The way he got the
data was through cores in boggy areas. The areas were sectioned,
and carbon was dated to assess when the sediment was laid down.
They also got botanists to determine what pollen was present, and
thus the types of vegetation that existed. So, they ran the
model and used these indicators to predict the weather, and hence
the vegetation.
The professor then told the audience what the model said
about the future: that the world would experience more weather
change in the next 100 years than it has in the last 20,000
years; that there will be an order of magnitude increase of
violent weather activity, which means ten times the number of
lightning strikes, ten times the number of hurricanes and
tornadoes, etc.; that the Great Lakes and other areas will become
dust bowls; that the change in weather will cause a
redistribution in rain, so more moisture is going to be taken out
of the soil than is replaced; that trees are going to be
increasingly set on fire by lightening; and that most of the
deciduous forests in the mid-west will burn to the ground.
I hasten to add that there are competent scientists who
think this is all nonsense...that there will be some mitigating
factor that will cause this not to occur. But, should we sit
around and wait to find out which rocket scientist is right?
We have to move post-haste to implement the kind of
monitoring systems on a global scale that will give us an early
warning and tell us who is right on this issue. It's not
something that we can sit around and guess about. We have to
find data that's credible, quality assured, and has been put
through models of verification.
56
-------
So, I think the type of activities that you and we at EPA
are engaged in are extremely important, not only to this country,
but to the globe at large. We need to implement quality
throughout our organization, and we need to take whatever effort
is needed to make EPA a quality organization."
57
-------
58
-------
QA MANAGER OF THE YEAR AWARD
-------
The QA Manager of the Year award is given to the individual
who demonstrates outstanding contributions in the field of
quality assurance. Sponsored by the Office of Research and
Development (ORD), the award consists of a plaque and a $2,000
cash award. Nominations are accepted by the Assistant
Administrator for ORD, and reviewed by a panel -of senior
managers.
The nominees for this year were Bill Laxton from the Office
of Air Quality Planning and Standards, who is noted for his
efforts in transferring the precision and accuracy data for the
ambient air monitoring networks from ORD databases to the ambient
monitoring network; Vicky Lloyd from the Office of Radiation
Programs, who has been instrumental in the EMMC activities on
methods integration and work on SW 846 waste methods, and acted
as liaison with the Department of Energy and interagency
agreement dealing with radioactive and mixed waste analytical
test methods; Lee Salmon and Phil Jalbert, also from the Office
of Radiation Programs, recognized for using TQM to develop a
successful national radiation measurement proficiency evaluation;
Kendall Young, recently retired from Region 6, known for his
contributions to the Drinking Water and other programs, and for
developing and presenting training programs for regional and
state employees; the Pretreatment Unit in the Water Division of
Region 6, noted for developing and implementing a pre-treatment
program audit process to ensure that the municipalities pre-
treatment program was functioning properly; and Bill Mitchell
from the Atmospheric Research and Exposure Assessment Laboratory
in Research Triangle Park, recognized for his involvement in
developing high quality control materials for QA training, and
for saving $2.8 million by redesigning an air monitoring study
and making sure that enough data was collected for an adequate
decision. Guy Simes and Barry Towns were nominated again this
year for continuing to help QAMS better understand the status of
programs across the agency, but Nancy Wentworth announced that
"until we reward all of the stellar performers, nobody gets it
twice."
Calling for a drum roll, Nancy Wentworth announced Kendall
Young as the 1991 QA Manager of the Year.
59
-------
60
-------
KENDALL YOUNG
1991 QA MANAGER OF THE YEAR
"To be honored by your peers is one of the greatest honors
you could give and I consider this one of the highest honors that
I've ever had. EPA has been a great life, and I certainly
appreciate all of the associations that I have had with so many
of you people here tonight. It has been well worth my life's
efforts and I hope that what I contributed was worth this honor.
I had a dream the other night about receiving a meshed bag
of gold dust, where the dust was gradually falling out. Maybe
there's some symbolism there which says that the environment is
in a meshed bag and is slowly falling out. That bag could
possibly be made more impervious by using all of our efforts to
protect the environment.
I'd like to share a vision with you, a vision of EPA: that
it will be called the Environmental Quality Protection Agency, a
multi-media organization with a water program, an air program, a
waste program, a toxic program, and all the other legislative
parameters that come along. I'd also like to see EPA make ethics
a cornerstone in the agency, and have the quality assurance
director report to the administrator. I'd like to envision that
there's a total commitment at EPA to the environment and guality
assurance, and that quality assurance and total quality
management not be a dichotomy. And lastly, I'd like to see the
fluff taken out of the agency's vernacular. For instance,
quality. What is quality? How much quality? That is too
generic a term. We need to talk in terms of standards and
parameters that define what quality is. I think that these
ambiguities can be removed. And for the nation, I'd like to
envision the President having a quality assurance person report
directly to him. I think that would be very useful to our
nation.
I thank you all and wish you well. I hope that the road
will rise up to meet you and the wind be always at your back."
61
-------
62
-------
TRAINING SESSIONS
-------
Four training sessions were offered on Wednesday, February 12
to help the conference participants enhance their technical and
management skills as QA professionals.
The Training Doctor Is In
Mary Ann Pierce, JWK International, Inc.
The purpose of this session was to give participants an overview of
the QA training system and to provide answers to specific QA
training problems.
The session began with a detailed examination of all available
Quality Assurance training courses and materials that have been
developed by QAMS during the past several years. The discussion
featured mini-lectures, video clips, and computer-based training
demonstrations. The second half of the workshop consisted of
participants creating their own individual training agendas,
tailored to the needs of their organizations and programs.
Career Skills and Strategy for Marketing QA
Joanne Jorz, Conceptual Systems, Inc.
This training session was designed to encourage attendees to focus
on their own strengths and development areas in the field of
quality assurance.
The session began with a brief presentation on the Meyers-Briggs
Type Inventory test and its relationship to the role of the Quality
Assurance officer. The concept of marketing was discussed briefly,
and participants were asked to assess their own ability to perform
specific QA marketing tasks. Based on their assessments, each
participant wrote an individualized career development plan to
address weaknesses and reinforce strengths.
The last portion of the session required participants to tailor
particular marketing strategies to a personal marketing plan for a
QA product or service.
Essential Skills for Change Agents: Change/ TQM, and Communications
Linne Bourget, Ph.D., Positive Management Communications
Systems
The purpose of this session was to provide insights and practical
skills for managing change with positive results. The workshop
focused on initiating changes involving QA and TQM within the EPA
organizational structure. Special emphasis was placed on
understanding customer needs, building relationships with
customers, communicating effectively, and understanding change
styles.
63
-------
Environmental Statistics for the QA Practitioner
This training session consisted of three sequential presentations
related to environmental statistics.
How Many Samples Do You Want Me To Take?
John Warren, Office of Policy Planning and Evaluation
This session was designed to familiarize even non-statisticians
with the different ways this question could be answered, and the
merits of various approaches such as: the historical variability
of the data (the variance or the standard deviation); using trial
samples to estimate variance; and using tolerance intervals.
Statistical Sampling Design
Robert O'Brien, Office of Policy, Planning and Evaluation
This presentation was targeted to participants whose statistical
background ranged from elementary to intermediate. A review was
given on frequently used approaches in developing sampling designs,
with an emphasis on designs tailored to support real world
decisions based on data collected. The primary components of the
sampling design were listed as: constructing the sampling frame,
sample selection procedures, estimation procedures, and procedures
for calculating sampling errors.
A Chemist Looks at Statistical Methods
Charles .Ramsey, National Enforcement Investigation Center
This session addressed problems encountered when applying
statistical techniques to environmental samples, methods for
subsampling, and quality control methods for sampling. Key
information included materials on sample design, quality control,
number of samples, and approaches to sampling design.
64
-------
OFFICE OF RESEARCH AND DEVELOPMENT (ORD)
Topics of discussion;
"Scientific Integrity:
Data Authenticity and Data Integrity"
"EPA and Contractors: Practical Considerations
for Implementing QA Programs"
"Revision of QAMS QA Documentation"
-------
"Scientific Integrity - Data Authenticity and Data Integrity"
The purpose of this session was to discuss and recommend
solutions for incorporating data authenticity into QAMS policy.
Jeffrey Worthington, Corporate Director of Quality Assurance,
TechLaw, Inc. gave a brief presentation on ensuring data
authenticity in environmental laboratories. He stressed the need
for laboratories and other generators of environmental data to
develop policies and procedures to ensure data authenticity as a
normal part of conducting business. While Worthington
acknowledged that there is no absolute method to guarantee data
authenticity, he noted several activities that environmental
professionals can perform to increase data authenticity:
1. Education and Training. A training program should be
developed that includes training schedules and goals for current
staff. The program should also be used for the orientation and
training of new staff.
2. Ethics and Data Integrity Agreement. Individuals who
sign this agreement will better understand the responsibilities
of reporting data to the public and private sector. This
document would focus responsibility on all staff members, not
just the final data reviewers. See Figure l on the next page.
3. Internal Audits. Internal audits should be conducted on
a routine basis. The quality assurance officer and field or
laboratory personnel should develop a plan for implementing
corrective action. After the plan is included in the internal
audit report, the quality assurance officer should review the
findings and ensure that corrections have been made.
4. Participation in Accreditation Program. Accreditation
officials would serve as third party auditors. There are two
advantages of using third party auditors. First, a third party
auditor may notice something an internal auditor does not.
Secondly, a third party auditor is more credible when showing
clients that the laboratory is reporting authentic data.
5. Quality Assurance Officer Observation Log. The quality
assurance officer should maintain a record of problems and
corrective actions taken as evidence of implementation.
6. Written Standard Operating Procedures (SOPs). SOPs
should be developed that describe the process for conducting
internal audits and outline a corrective action plan. These
procedures should be provided to the client.
7. Documentation Procedures. Environmental professionals
should develop and enforce procedures that ensure that
information is not recorded on temporary records for later
transfer to a "neat" field record or laboratory benchsheet.
65
-------
8. Automated Laboratory Practices. Many laboratories use
the laboratory management system (LIMS) for recording data.
Guidance on the procedures for entering information into these
systems is available in draft guidance by the EPA Office of
Information Resources Management (OIRM). Areas included in the
guidance are: quality assurance of software, ensuring the
accuracy of data entry, lifecycle documentation of software
development, definition of computer raw data, traceability of
data edits, and physical security and tiered access
considerations.
9. Data Certification by the Laboratory. Formal data
validation, which is similar to a certification process, should
verify that data meets contractual requirements and is authentic.
10. Usability Determination. Environmental data users
should verify data authenticity as an informal part of the
usability determination process. This process can be facilitated
by providing good evidence of data accuracy in data packages.
Figure 1: Ethics and Data Integrity Agreement
ETHICS AND DATA INTEGRITY AGREEMENT
1. 2, a^mfi. state that I understand the high standards
of integrity required of me with regard to the duties I perform and the data I report
in connection with my employment at irmmni
I agree that in the performance of my dunes at Hum i
a. I shall not intentionally report data values thai are not the actual values obtained;
b. I shall not intentionally upon the dales and times of dan analyses mat are not
the actual dates and times of daa analyses; and
c. I shall not intentionally represent another individual's work as my own.
HI. I agree to inform fKnimn of any
acodenta! reporting of non-aomemic daa by myself in a timely manner.
IV. I agree to inform nunniiini -*—y
accidental or intentional reporting of non-authentic data by other employees.
66
-------
"EPA and Contractors: Practical Considerations for Implementing
QA Programs"
The goal of this session was to examine the ORPM Transmittal
92-13, entitled "Quality Assurance Activities in ORD's Contract
Research," and discuss concerns about contractor personnel
performing QA oversight activities that may involve their own
companies or their competitors. The outcome was a list of
potential problems and recommendations compiled by ORD QA
Managers regarding ORPM Transmittal 92-13. The five key points
discussed were:
1. The Project Officer is the final authority on accepting
the quality of a product, but must consult the QA program when
environmental measurement data are involved (i.e standards for
the data quality objective process (DQO) and requirements of the
quality assurance project plan (QAPjP).
2. A contractor cannot evaluate the work of another
contractor who is a competitor or potential competitor without
proper controls. Controls in a contractor/contractor evaluation
should include tl>a opportunity for the reviewed contractor to
check the evaluation for accuracy, and enable an EPA employee to
examine and approve the report for accuracy and bias.
3. Any contract under $50,000 that has QA support services
within its statement of work should require justification for on-
site evaluations.
4. Self-evaluation by the contractor should not be the only
criteria for judging product adequacy. Controls for contractor
self-evaluation should include the use of predefined criteria; a
requirement that QA personnel report to a different
organizational unit than technical personnel; and a requirement
that QA personnel certify that they will perform evaluations in
an impartial and ethical manner. All controls must be overseen
by an EPA employee.
5. The group was asked to complete a survey on the types of
activities performed by QA contractors with and without EPA
oversight. Figure 2 shows the results of the survey.
"Revision of QAMS QA Documentation"
The goal of this session was to analyze current guidance
documents for appropriate revisions and new guidance. Members of
the group reviewed existing documents and discussed future ORD QA
guidance needs. Three key questions were identified: Which
current documents are being used? What improvements are needed
in current documents? What additional guidance is needed for
ORD? Among the needs addressed were:
1. More guidance on data review/data validation
2. A quick, computerized system for QAPP development
3. Development of training and education programs
4. To make clear who needs it, and why and how it "serves
the science"
5. Reassess Standard Operating Procedures
67
-------
Figure 2: Survey Results on Activities Performed by QA
Contractors with and Without EPA oversight
Activity
EPA Staff Contractors Contractors
w/Oversight w/o Oversight
1 . Review of QA planning
documents.
2. Review of final reports.
3. Conduct of audits, including
assessing the quality of field
studies or laboratory studies
or modeling research.
4. Assistance in preparing QA
planning documents.
5. Preparation of QA/QC
materials for
meetings/symposiums.
6. Presentation of QA/QC
materials at meetings and
symposiums.
7. Literature search relative to
QA/QC Issues
8. Preparation and presentation
of relevant QA training
courses and material.
8. QA tracking and reporting.
10. Experimental design
development
1 1 . Experimental design review.
12. Project management of
sub-tasks.
7
5
6
7
7
6
5
6
5
4
6
6
5
5
7
5
6
.6
4
7
4
3
6
3
0
0
0
0
1
2
2
0
0
0
0
0
68
-------
NATIONAL PROGRAM OFFICES
Topics of discussion;
"The Scope of Quality Assurance at EPA HQ:
Current Challenges/Future Extensions"
"HQ QA Group Expectations of QAMS/QAMS Expectations
of the HQ QA Group"
-------
"The Scope of Quality Assurance at EPA HQ: Current
Challenges/Future Extensions"
The purpose of this session was to discuss current and
future challenges of the EPA Order and identify possible
improvements. The group focused on several key.questions: What
are the primary current challenges for viable QA in the current
scope of the EPA Order? What are the overall Agency needs in QA
and how are they being met? Should the Agency expand the QA
function beyond "environmental data?" Who should be responsible
for any identified expansions of QA beyond "environmental data?"
By the end of the session, the group had compiled a list of
current challenges. These included:
1. Increase QA visibility and assure suitable priority for
QA
a. Better educate management on QA Manager's role
b. Better coordinate and define Quality Assurance roles
c. Better involvement in engineering processes
d. Better involvement in regulation development
2. Improve communication and linkages
a. Hold more meetings between Headquarters and QA
Managers
b. Formalize the status of HQQAG meetings
c. Better define jurisdiction of QA activities
d. Identify overlapping areas
e. Define linkages of Headquarters QA Managers to
cross-cutting program offices and groups like OIRM,
OPPE, CES, etc.
3. Expand QA scope beyond current narrow definition
a. Include non-environmental data (i.e Administrative
data)
4. Close the QA assessment loop so feedback on data quality
gets to the Headquarters QA Manager for evaluation
5. Improve the OA for compilation of data
a. Better interagency data
b. Better indicators and indices
6. Improve interaction on Information Management QA needs
7. Identify impacts on HO OA activities from ANSI/ASQC-E4
and potential update of the EPA Order 5360.1
69
-------
"HQ QA Group Expectations of QAMS - QAMS Expectations of the HQ
QA Group"
This session was designed to familiarize participants with
the HQ QA group and how it interacts with QAMS. Group members
discussed and compiled a list of HQQAG's expectations of QAMS.
Some of the expectations included:
1. Leadership
a. Ensure that QA officers are at proper level,
visible, have access to senior managers
b. Ensure that QA officers have sufficient time to
perform QA activities
c. Work with OIRM and OPPE to get QA into Agency-wide
strategic planning, data integration, and
environmental indicators
d. Develop an effective strategy for the next 3-5 years
2. Customer-oriented support
a. Develop effective Headquartprs QA Team/Network for
frequent communication on substantive issues and
effective peer assistance across program lines
b. Promote linkage between different offices and cross-
program solutions as often as possible
3. Guidance
a. Include Headquarters and Regional QA managers
earlier and more frequently in developing and
reviewing policies and guidance
b. Expect QAMS to reflect program needs and
capabilities in its guidance
c. Develop a process/mechanism to ensure coordination
between QA and those developing policy, regulations,
and Agency guidance
4. Increase Technical Assistance
5. Training
a. Develop a QA curriculum
b. Establish a training fund
70
-------
REGIONAL PROGRAM OFFICES
Topics of discussion;
"QA in Regulatory Development"
"QA Management Plans and FMFIA Issues"
"Development of a Model Management Systems Audit (MSA)
Format for the Removal and Emergency Preparedness Programs"
"Drinking Water Monitoring Quality Issues"
"Documentation Requirements for Data Produced by
Non-CLP Laboratories"
-------
"QA in Regulatory Development"
This session focused on the regional role in the regulatory
development process. Group members discussed current activities
and objectives for maximizing regional participation in
Regulatory Development Quality Teams. Several suggestions for
improving region involvement included: 1. generating better
documentation for QA within the Agency and 2. having documents
that are accurate and implemented, rather than documents that are
in-hand.
The Start Action Notice form and the effects of a regulatory
moratorium were briefly discussed. The process was described as
"in place," with relevant forms currently available.
The outcome of the session was a greater awareness of
regional opportunities to participate in the regulatory
development process and a greater awareness of regional issues
important to the Quality Action Team.
"QA Management Plans and FMFIA Issues"
This session was designed to update the group on current
political and management views of QA documentation. A short
presentation was given on FMFIA designation of QA documentation
as an Agency-wide weakness and the role that QA plays in response
to the designation. Key questions addressed were: Why use FMFIA
now? Is this a bureaucratic response to the beancounters? Will
this make any difference?
"Development of a Model Management Systems Audit (MSA) Format for
the Removal and Emergency Preparedness Programs"
The purpose of this session was to examine Region II's case
study in developing a Management Systems Audit format as a pilot
for use by other regions and adopt a format for conducting MSAs
on the Removal and Emergency Response components of the Superfund
program. The first half of the discussion focused on the process
and materials used to develop the format for an MSA; and the
second half focused on the effectiveness of the MSA's content,
format, and applicability in other regions. Key questions
included: Does the audit content adequately address all issues
of concern? Can the audit format be improved? Is the model
format easily implemented?
"Drinking Water Monitoring Quality Issues"
This session began with a short presentation on the
structure, participants, and accomplishments of the Drinking
Water Quality Monitoring Work Group (DWQMWG). The discussion
centered on quality-related issues being addressed by the DWQMWG,
71
-------
and the draft Federal Register notice on the quantification of
drinking water analyses. The outcome of the discussion was a
list of comments and suggestions on the effectiveness of DWQMWG's
proposals.
"Documentation Requirements for Data Produced by Non-CLP
Laboratories"
The group examined the requirements specified for Superfund
CLP laboratories under CERCLA to determine their applicability to
Non-CLP laboratories. A brief presentation on the history of the
Superfund Contract Laboratory Program was given, along with a 30-
minute talk on "Documentation Requirements for Analytical Data"
by Region 10. The goal of the discussion was to produce a
National Guidance document that could be referenced in Quality
Assurance Project Plans and given to any laboratory producing
environmental monitoring data under such plans.
72
-------
JOINT SESSION (ORD, RPO, NPO)
Topics of discussion:
"Quality Assurance in Model Development,
Computer Software, and Electronic Data Transfer"
-------
"Quality Assurance in Model Development, Computer Software, and
Electronic Data Transfer"
The session began with a presentation by Rick Johnson on a
recent review conducted on the capacity of EPA's information
resources. He addressed the historical perspective and the
findings and recommendations from the review. The group then
separated into three sub-groups to identify the problems and
issues regarding inadequate QA/QC in the areas of model
development, software development, electronic data transfer, and
electronic recordkeeping. The list included the following:
Model Development
o Applications limits not given
o Assumptions not documented
o Sensitivity analysis not performed
o Required time and spatial scales not clearly specified
o Generated output precision often missing
o Hardware requirements for running model not clear
Software Development
o Documentation often missing or incomplete
o Failure to debug before release
o Failure to assess or consult user's capabilities
o Failure to adequately specify performance criteria
o Failure to use case tools
o Re-invent software testing procedures
Electronic Data Transfer
o Lack of specification for transferred data
o Incompatibility with other systems
o Transferred data not checked for completeness or
defects
Electronic Recordkeeping
o Undocumented changes, loss of traceability
o Data quality screens
o Legally defensible? Authentic?
o Access and security?
o Failure to backup
o Need to reformat to keep up with current technology
The group then recommended ten action items for resolving
the issues and problems identified:
1. Gather literature on model QA/QC
2. Develop standard for format and transmission
3. Establish standards and requirements for model
evaluation
4. Develop/review/archive appropriate documentation
5. Conduct performance evaluations using valid data sets
6. Develop agency policy to address:
73
-------
a. Model/software QA/QC
b. Inherent error in models
c. Validating assumptions
d. Minimum data entry requirements
7. Give results of this session to QAMS for follow-up with
OIRM committees and work-groups
8. Require progress report from QAMS/OIRM at 13th annual
meeting
9. Make related guidance, standards, procedures available
as part of proceedings
10. Supply list of participants as potential contacts for
work-groups or committees
74
------- |