ENVIRONMENTAL
                              PROTECTION
                               AGENCY

                             DALLAS, TEXAS

                              LIBRARY
GUIDANCE FOR THE DATA QUALITY
        OBJECTIVES PROCESS
             EPA QA7G-4
    United States Environmental Protection Agency
        Quality Assurance Management Staff


            Washington, DC 20460
                  FINAL


             SEPTEMBER 1994
                                      Printed on Recycled Paper

-------
                                   FOREWORD
       The U.S. Environmental Protection Agency (EPA) has developed the Data Quality
Objectives (DQO) Process as an important tool for project managers and planners to
determine the type, quantity, and quality of data needed to support Agency decisions.  This
guidance is the culmination of experiences in applying DQOs in different Program Offices at
die EPA.  Many elements of prior guidance, advice, statistics, and scientific planning have
been incorporated into this document.  This guidance supersedes all previous guidance,
including the EPA's "Development of Data Quality Objectives, Description of Stages I and
II" (July 1986), and "Guidance for Planning for Data Collection in Support of Environmental
Decision Making Using the Data Quality Objectives Process" (Interim Final, October 1993).
This document is consistent with the Office of Emergency and Remedial Response guidance,
"Data Quality Objectives for Superfund" (EPA 540-R-93-071).

       The purpose of this document is to provide general guidance to organizations on
developing data quality criteria and performance specifications for decision making. This
guidance assumes that an appropriate Quality System has been established and is operational.

       This guidance  has been prepared in response to EPA Order 5360.1, entitled "Policy
and Program  Requirements to Implement the Quality Assurance Program," which establishes
requirements  for quality assurance when generating environmental data in support of Agency
decisions. In addition, this guidance reflects the policy of the Agency to develop and
implement the DQO Process as expressed by Deputy Administrator A. James Barnes in his
memorandum on  "Agency Institutionalization of Data Quality Objectives," dated November
1986.

       This document is a product of the collaborative effort of many quality management
professionals  throughout the EPA and among the contractor community. It has been peer
reviewed by the EPA  Program Offices, Regional Offices,  and Laboratories. Many valuable
comments and suggestions have been incorporated to make it more useful.
EPAQA/G-4                                  ii         '                    September 1994

-------

-------
                                Table of Contents



Chapter                                                                         Page

       Foreword	  ii

       List of Figures and Tables  	iv

       Introduction	  1

1.     Step 1:  State the Problem  	  9

2.     Step 2:  Identify the Decision	   13

3.     Step 3:  Identify the Inputs to the Decision	   17

4.     Step 4:  Define the Boundaries of the Study	   19

5.     Step 5:  Develop a Decision Rule	  23

6.     Step 6:  Specify Tolerable Limits on Decision Errors	  27

7.     Step 7:  Optimize the  Design for Obtaining Data  	  37

       Bibliography	  41


Appendices

A.     Beyond the DQO Process: The Quality Assurance Project Plan and
       Data Quality Assessment  	  43

B.     DQO Case Study:  Cadmium-Contaminated Fly Ash Waste	  47

C.     Derivation of Sample  Size Formula for Testing Mean of Normal
       Distribution Versus an Action Level	  61

D:     Glossary of Terms	  65
EPA QA/G-4                                 iii                            September 1994

-------
                                  List of Figures
                                                                                Page
0-1.   The Data Quality Objectives Process  .	  2

0-2.   Repeated Application of the DQO Process Throughout the
      Life Cycle of a Single Project	  5

2-1.   Example of Multiple Decisions Organized Into a Flowchart	  16

4-1.   An Example of How to Stratify a Site With Soil Contamination	  22

6-1.   An Example of a Decision Performance Goal Diagram —
      Baseline Condition:  Parameter Exceeds Action Level	  35
                                                                t

6-2.   An Example of a Decision Performance Goal Diagram —
      Baseline Condition:  Parameter is Less Than Action Level	'. .  36

7-1.   An Example of a Power Curve —
      Baseline Condition: Parameter is  Less Than Action Level 	  40

A-l.  QA Planning and the Data Life Cycle 	  44

A-2.  Quality Assurance Assessment  	  46       ™

B-l.  Design Performance Goal Diagram for Cadmium Compliance Testing —
      Baseline Condition:  Mean Exceeds Action Level	  53
                                  List of Tables


1-1.   Elements of the Problem Description	  12

5-1.   Attributes of Different Statistical Parameters to Characterize the Population	  25

6-1.   Decision Error Limits Table Corresponding to Figure 6-1	  35

6-2.   Decision Error Limits Table Corresponding to Figure 6-2	 .  36
EPAQA/G-4                                 IV         '                   September  1994

-------
                                 INTRODUCTION
       Each year the U.S. Environmental Protection Agency (EPA) and the regulated
community spend approximately $5 billion collecting environmental data for scientific
research, regulatory decision making, and regulatory compliance.  While these activities are
necessary for effective environmental protection, it is the goal of EPA and the regulated
community to minimize expenditures related to data collection by eliminating unnecessary,
duplicative, or overly precise data. At the same time, the data collected should have
sufficient quality and quantity to support defensible decision making.  The most efficient way
to accomplish both of these goals is to establish criteria for defensible decision making before
the study begins, and then develop a data collection  design based on these criteria. To
facilitate this approach, the Quality Assurance Management Staff (QAMS) of EPA has
developed the Data Quality Objectives (DQO) Process, a systematic planning tool based on
the Scientific Method for establishing criteria for data quality and for developing data
collection designs. By using the DQO Process to plan environmental data collection efforts,
EPA can improve the effectiveness, efficiency, and defensibility of decisions hi a resource-
effective manner.

What are DQOs?  DQOs are qualitative and quantitative statements derived from the outputs
of the first six steps of the DQO Process that:

       1)     Clarify the study objective;

       2)     Define the most appropriate type of data to collect;

       3)     Determine the most appropriate conditions from which to collect the data; and

       4)     Specify tolerable limits on decision errors which will be used as the basis for
              establishing the quantity and quality of data needed to support the decision.

The DQOs are then used to develop a scientific and  resource-effective data collection design.

What is the DQO Process?  The DQO Process is a strategic planning approach based on the
Scientific Method that is used to prepare for a data collection activity.  It provides a
systematic procedure for defining the criteria that a data collection design should satisfy,
including when to collect samples, where to collect samples, the tolerable level of decision
errors for the study, and how many samples to collect.

       By using the DQO Process, the Agency will assure that the type, quantity,  and quality
of environmental data used in decision making will be appropriate for the intended
application.  In addition, the  Agency will guard against committing resources to data
collection efforts that do not support a defensible decision.
EPAQA/G-4                                   1         '                    September 1994

-------
       The DQO Process consists of seven steps, as shown in Figure 0-1.  The output from
each step influences the choices that will be made later in the Process.  Even though the DQO
Process is depicted 33 a linear sequence of steps, in practice it is iterative; the outputs from
one step may lead to reconsideration of prior steps. This iteration should be encouraged since
it will ultimately lead to a more efficient data collection design.  During the first six steps of
the DQO Process, the planning team will develop the decision performance criteria (DQOs)
that will be used to develop the data collection design.  The final step of the Process involves
developing the data collection design based on the DQOs. The  first six steps should be
completed before the planning team attempts to develop the data collection design because
this final step is dependent on a clear understanding of the first  six steps taken as a whole. In
Figure 0-1, the iterative link between the DQOs and the Optimize the Design step is
illustrated by double  arrows, which signify that it may be necessary to revisit any one or
more of the first six steps  to develop a feasible and appropriate  data collection design. Above
all, every step should be completed before data collection begins.

State the Problem
*
Identify the Decision
*
Identify Inputs to the Decision
*
Define the Study Boundaries
*
Develop a Decision Rule
*
Specify Limits on Decision Errors

                                  ±±
               Optimize the Design for Obtaining Data
              Figure 0-1. The Data Quality Objectives Process.
EPA QA/G-4
September 1994

-------
       Each of the seven steps is described briefly below.  A more detailed description can be
 found in the subsequent chapters of this guidance.

       •   Step 1:  State the Problem —  Concisely describe the problem to be studied.
           Review prior studies and existing information to gain  a sufficient understanding to
           define the problem.

       •   Step 2:  Identify the Decision —  Identify what questions the study will attempt
           to resolve, and what actions may result.

       •   Step 3:  Identify the Inputs to the Decision —  Identify the information that needs
           to be obtained and the measurements that need to be taken to resolve the decision
           statement.

       •   Step 4:  Define the Study Boundaries  — Specify the time periods and spatial
           area to which decisions will apply.  Determine when and where data should be
           collected.

       •   Step 5:  Develop a Decision Rule  — Define the statistical parameter of interest,
           specify the action level, and integrate the previous DQO outputs into a single
           statement that describes the logical basis for choosing  among alternative actions.

       •   Step 6:  Specify Tolerable Limits on Decision Errors  — Define the decision
           maker's tolerable decision error rates1 based on a consideration of the
           consequences of making an incorrect decision.

       •   Step 7:  Optimize the  Design — Evaluate information from the previous steps
           and generate alternative data collection  designs.   Choose the most resource-
           effective design that meets all DQOs.

Who should read the DQO guidance? This guidance is intended for project managers and
other members of a planning team that will use the DQO Process to structure the  data
collection planning process and to develop an appropriate data collection design.   In addition,
the guidance may be relevant to other staff members who will participate in the study.
Consult with an EPA Quality Assurance Manager,  Quality Assurance Officer, or Quality
Assurance  Representative to obtain additional advice on who should read this guidance.
   1 A decision error rate is the probability of making an incorrect decision based on data that inaccurately
estimate the true state of nature.

EPAQA/G-4                                   3         '                     September 1994

-------
What projects are covered by this guidance? This guidance document covers all projects
where:
       1) the objective of the study is to collect environmental data in support of an Agency
          program, and

       2) the results of the study will be used to make a specific decision.

Every step of this guidance may not be applicable to data collection activities where specific
decisions cannot be identified, such as studies that are exploratory hi nature. The reason for
this distinction is that part of the DQO Process includes formulating statistical hypotheses.  If
a statistical hypothesis is not linked to a clear decision in which the decision maker can
identify potential consequences of making a decision error, then some of the activities
recommended in this guidance may not apply.  Nonetheless, the DQO Process is still a
valuable tool that can be used to help plan studies where the data are not directly used to
support a specific decision.  In these cases, it may be possible to frame a research type study
question hi the form of a decision or modify the activities described in this guidance to
address the needs of the study.

What is the value of using the DQO Process?

       •  The DQO Process is a planning tool that can save resources by making data
          collection operations more resource-effective.  Good planning will streamline the
          study process and increase the likelihood of efficiently collecting appropriate and
          useful data.

       •  The structure of the DQO Process provides a convenient way to document
          activities and decisions and to communicate the data collection design to others.

       •  The DQO Process enables data users and relevant technical experts to participate
          in data collection planning and to specify then- particular needs prior to data
          collection.  The DQO process fosters communication among all participants, one
          of the central tenets of quality management practices.

       •  The DQO Process provides a method for defining decision performance
          requirements that are appropriate for the intended use of the data.  This is done by
          considering the consequences of decision errors and then placing tolerable limits
          on the probability that the data will mislead the decision maker into committing a
          decision error.  A statistical sampling design can then be generated to provide the
          most efficient method for controlling decision errors and satisfying the DQOs.

       •  The DQO Process helps to  focus studies by encouraging data users to clarify
          vague objectives  and to limit the number of decisions that will  be made.
EPAQA/G-4                                   4         '                     September 1994

-------
When should the DQO Process be used?  The DQO Process should be used during the
planning stage of any study that requires data collection, before the data are collected.  In
general, EPA's policy is to use the DQO Process to plan all data collection efforts that will
require or result in a substantial commitment of resources.  The Quality Management Plans
(QMPs) of the Agency's National Program Offices, Regional Offices, and Research and
Development organizations will specify which studies require DQOs.

Can the DQO Process be used for small studies? The DQO Process applies to any study,
regardless of its size.  However, the depth and detail of DQO development will depend on the
complexity of the  study. The more complex a study, the more likely that it will have several
decisions that could benefit from the DQO Process and that the decisions will require more
intensive DQO development.

Should the DQO  Process be applied as intensively to all situations? No, the DQO Process
is a flexible planning tool that can be used more or less intensively as the  situation requires.
For projects that have multiple decisions, where the resolution of one decision only leads to
the evaluation of subsequent decisions, .the DQO Process can be used repeatedly throughout
the life cycle of a  project.  Often, the decisions that are made early in the project will be
preliminary in nature.  They might require only a limited planning and evaluation effort. As
the study nears conclusion and the possibility of making a decision error becomes more
critical, however, the level of effort needed to resolve a decision generally will become
greater.  Figure 0-2 illustrates this point.
                                  STUDY PUNNING
                                   COMPLETED
                                                     STUDYPtANNNa
                                                       COMPLETED
                                                                           STUDY PIANNINQ
                                                                            COMPLETED
                          INCREASING LEVEL OF EVALUATION EFFORT
 Figure 0-2.  Repeated Application of the DQO Process Throughout the Life Cycle of a
                                    Single Project
EPA QA/G-4
September 1994

-------
Who participates in the DQO Process?  A DQO planning team generally consists of senior
program staff, technical experts, senior managers, someone with statistical expertise, and a
Quality Assurance (QA)/Quality Control (QC) advisor, such as a QA Manager. It is
important that all of these people, including managers, participate (or stay informed) from the
beginning of the DQO Process so that it can proceed efficiently.   .

What are the outputs of the DQO Process?  The DQO Process leads to the development of
a quantitative and qualitative framework for a  study.  Each step of the Process derives
valuable criteria that will be used to establish the final data collection design.  The first five
steps of the DQO Process identify mostly qualitative criteria such as what problem has
initiated the study and what decision it attempts to resolve. They also define the type of data
that  will be collected, where and when the data will be collected, and a decision rule that
defines how the decision will  be made.  The sixth step defines quantitative criteria expressed
as limits on decision errors  that the decision maker can tolerate. The final step is used to
develop a data collection design based on the criteria developed in the first six steps.  The
final product of the  DQO Process is a data collection design that meets the quantitative and
qualitative  needs of the study.                                      ,

       Much of the information that is developed in the DQO Process will also be useful for
the development of  Quality Assurance Project  Plans (QAPPs) and the implementation of the
Data Quality Assessment (DQA) Process. The outputs of the DQO Process can be used
directly and indirectly as inputs to a QAPP. To evaluate the data using the DQA Process, it
is necessary to have first established decision quality criteria using the DQO Process or its
equivalent.  Therefore, the DQO  Process not only helps plan a study, establish decision
quality criteria, and  develop a data collection design, but it also aids in the development of
QAPPs and the DQA  Process.

What is a  data collection design?  A data collection design specifies  the final configuration
of the environmental monitoring or measurement effort required to satisfy the DQOs.  It
designates the types and quantities of samples  or monitoring information  to be collected;
where, when, and under what conditions they should be collected; what variables are to be
measured; and the QA/QC procedures to ensure that sampling design and measurement errors
are controlled  sufficiently to meet the tolerable decision error rates specified in the DQOs.
These QA/QC procedures are established in the QAPP.

Where does the DQO Process fit into EPA's Quality System? The DQO Process is the
part  of the  Quality System that provides the basis for linking the intended use of the data to
the QA/QC requirements for data collection and analysis.  This document is one of a series of
quality management requirements and guidance documents that the U.S. EPA Quality
Assurance Management Staff (QAMS) has prepared to assist users in implementing the
Agency-wide Quality  System.  The current document list contains:

EPA QA/R-1  EPA  Quality System Requirements for Environmental Programs

EPA QA/G-1  Guidance for Developing, Implementing, and Evaluating Quality Systems for
              Environmental  Programs
EPA QA/G-4                                   6                             September 1994

-------
EPA QA/R-2  EPA Requirements for Quality Management Plans

EPA QA/G-2  Guidance for Preparing Quality Management Plans for Environmental
              Programs

EPA QA/G-4  Guidance for The Data Quality Objectives Process

EPA QA/R-5  EPA Requirements for Quality Assurance Project Plans for Environmental
              Data Operations

EPA QA/G-5  Guidance for Quality Assurance Project Plans

EPA QA/G-9  Guidance for Data Quality Assessments

       Agency policy statements are found in the requirements documents (QA/R-xx series).
Advisory papers are found in the guidance documents (QA/G-xx series).
                                                                  i
Can existing data be used to support decisions using the DQO Process?  Existing data can
be very useful for supporting decisions using the DQO Process.  There are three ways that
existing data can be used:

       1)  If sufficient documentation is available, existing data may be used alone or
           combined with new data.  Determining whether data can appropriately be
           combined can be a very complex operation that should be undertaken with great
           care.  In many cases it will require the expertise of a statistician.

       2)  The existing data may provide valuable information (such as variability) that can
           be used in the development of the data collection design.

       3)  The existing data may be useful  in guiding the selection of an efficient data
           collection design.

Will the use of the DQO Process always result in statistical/probabilistic sampling
methods for data collection? No.  While statistical methods for developing the data
collection design  are strongly encouraged, this guidance recognizes that not every problem
can be evaluated using probabilistic techniques.  The DQO Process, however, can and should
be used as a planning tool for studies even if a statistical data collection design ultimately
will not be used.  In these cases, the planning team is encouraged to seek expert advice on
how to develop a non-statistical data collection design and on how to evaluate the result of
the data collection. When non-probabilistic, judgemental, or quota sampling methods are
used, be sure to consult with an EPA QA Manager, QA Officer, or QA Representative to
ensure that program-specific QA requirements are satisfied.

How should this guidance be used?  This  guidance should be used as a tool to structure the
planning activities for collecting  environmental data.  It should be used to organize meetings,
focus the collection of background information, and facilitate communication between
technical experts, program managers, and decision makers,  .

EPA QA/G-4                                   7                             September  1994

-------
How is this guidance structured? This guidance contains seven chapters, four appendices,
and a bibliography.  Each of the remaining chapters describes one of the seven steps of the
DQO Process. Each chapter is divided into four sections as follows:

       (1) Purpose - This section explains the objective of the chapter.

       (2) Expected Outputs - This section identifies the products expected upon completion
          of the DQO Process step.

       (3) Background - This section provides background information on the DQO Process
          step, including the rationale for the activities in  that step.

       (4) Activities - This section describes the activities  recommended for completing the
          DQO Process step, including how inputs to the step are used.

       Appendix A provides a brief overview of both the Quality Assurance Project Plan
(QAPP) development process, which is used to document the operational and QA/QC
procedures needed to implement the data collection design, and the Data Quality Assessment
(DQA) Process, which is used after the data have been collected to evaluate whether the
DQOs have been satisfied.  Appendix B is  a case study in which the DQO Process is applied
to an environmental problem. Appendix C provides a derivation of the sample size formula
used in Appendix B.  Appendix D provides a glossary of terms used in this guidance.

Where is it possible to get statistical support? Access to statistical support is available
through the EPA Quality Assurance Management Staff (QAMS) at (202) 260-5763.

How long will this  guidance be hi effect?  This guidance will remain in effect for five years
from the publication date, unless superseded by an updated version.

Where is it possible to get more information about the DQO Process?  A DQO training
course is available through the EPA at the U.S. EPA Headquarters in Washington, D.C.
Additional documents  on DQO applications can be obtained from the Quality Assurance
Management Staff at EPA Headquarters.

Two documents that can provide additional detail on the DQO Process are:

       •   U.S. Environmental Protection Agency.  1993. Data Quality Objectives Process
          for Superfund:  Interim Final Guidance. EPA 540-R-93-071.

       •   Bates, D.J., R.O. Gilbert, N.L. Hassig, R.F. O'Brien, B.A. Pulsipher, 1993.
          Decision Performance Criteria:  The Driver Behind The Data Quality Objectives
          Process — A Statistical Introduction (Draft).  Pacific Northwest Laboratory,
          Richland, Washington.
EPAQA/G-4                                   8                             September 1994

-------
                                   CHAPTER 1
                      STEP1:  STATE THE PROBLEM
 THE DATA QUALITY OBJECTIVES PROCESS


—
State the Problem
\ *
identify the Decision
\*
Identify Inputs reahe Decision
* \
Define the Study Boundaries
* N
Develop a Decision Rule
*
Specify Limits on Decision Errors
^M
\
s

          Optimize the Design for Obtaining Data
                                                     STATE THE PROBLEM
                                                   Purpose
                                                   To dearty define the problem so that the
                                                   focus of the study will be unambiguous.
                                                   Activities
                                                   • Identify members of the planning team.
                                                   • Identify the primary decision mater.

                                                   • Devetop a concise description of the probtem.

                                                   • Specify available resources and relevant
                                                    deadlines for the study.
Purpose

      The purpose of this step is to define the problem so that the focus of the study will be
unambiguous.

Expected Outputs

      •   A list of the planning team members and identification of the decision maker.

      •   A concise description of the problem.

      •   A summary of available resources and relevant deadlines for the study.
EPA QA/G-4
September 1994

-------
Background

       The first step in any decision making process is to define the problem that has
initiated the study.  Since most environmental problems present a complex interaction of
technical, economic, social, and political factors, it is critical to the success of the process to
define the problem completely and in an uncomplicated format.  A problem will have the
greatest chance of being solved when a multidisciplinary team of technical experts and
stakeholders can help to recognize all of the important facets of the problem and ensure that
complex issues are described accurately.  Generally teams will function more effectively
when they have one clearly identified decision maker.

       This step in the DQO Process addresses development of a planning team that will
define the problem and implement subsequent steps of the Process. It also calls for the
identification of a decision maker who will lead the planning team and make  final resolutions
during the Process.  The goal is to create a well-structured planning team that will work
effectively and efficiently to develop a concise and complete description of the  problem,
which will provide the basis for the rest of the DQO development.

Activities

Identify members of the planning team.  The planning team is the group that will develop
DQOs for the study.  The number of planning team members will be directly related to the
size and complexity of the problem. The team should include representatives from all groups
who are stakeholders in the project, including, but not limited to,  samplers, chemists and other
scientists and engineers, modelers, technical project managers, community representatives,
administrative and executive managers, QA/QC experts (such as a QA Manager), data users,
and decision makers. A reasonable effort should be  made to include any decision makers
who may use the study findings later.  A  statistician (or someone knowledgeable and
experienced with environmental statistical design and analysis) should also be included on this
team.

Identify the primary decision maker of the planning team and define each member's
role and responsibility during the DQO Process. The planning team generally has a leader,
referred to as the "decision maker." The  decision maker has the ultimate authority for
making final decisions based on the recommendations of the planning team.  The decision
maker is often the person with the most authority over the study,  and may be responsible for
assigning the roles and responsibilities to the planning  team members.  In  cases where the
decision maker cannot attend DQO planning meetings, a senior staff member should keep the
decision maker informed of important planning  issues.
EPAQA/G-4                                   10                             September 1994

-------
Develop a concise description of the problem. The problem description provides
background information on the fundamental issue to be addressed by the study.  Below is a
list of steps that may be helpful during this phase of DQO development.

       •   Describe  the conditions or circumstances that are causing the problem and the
           reason for understanding the study. Typical examples for environmental problems
           include conditions that may pose a threat to human health or the environment, and
           circumstances of potential non-compliance with regulations.

       •   Describe  the problem as it is currently understood by briefly summarizing, existing
           information. (See Table 1-1 for a list of elements that may be appropriate to
           include in the problem description.)

       •   Conduct literature searches and examine past or ongoing studies to ensure that the
           problem is correctly defined and has not been solved previously.  Organize and
           review relevant information, including preliminary studies, and indicate the source
           and reliability of the  information.  Take note of information  about the performance
           of sampling and analytical methods observed in similar studies since this
           information may prove to be particularly valuable later hi the DQO Process.

       •   If the  problem is complex, consider breaking it into more manageable pieces.
           Identify those pieces  that could be addressed by separate studies.  Assign priorities
           to and logical relationships among the pieces of the problem.

Specify the available resources and relevant deadlines for the study. Stipulate the
anticipated budget, available personnel, and contractual vehicles (if appUcable). Also,
enumerate  any  deadlines for completion of the study  and any intermediate deadlines that may
need to be met.
EPAQA/G-4                                   11         •                    September 1994

-------
                      Elements of the Problem Description
     The following elements may be appropriate to include in the problem description.
     Note: this list only provides the basic elements of the problem description.  Your
     elements may be slightly different.

            •   Study objectives/regulatory context.

            •   Persons or organizations involved in the study.

            •   Persons or organizations that have an interest hi the study.

            •   Political issues surrounding the study.

            •   Sources and amount of funding.

            •   Previous study results.

            •   Existing sampling  design constraints (some aspects of sampling design
               may be specified in regulations or established through past planning
               efforts).
                    Table 1-1. Elements of the Problem Description.
EPA QA/G-4
                                           12
September  1994

-------
                                     CHAPTER 2

                      STEP 2:  IDENTIFY THE DECISION
   THE DATA QUALITY OBJECTIVES PROCESS
              Identffyvlnputs to the Decision
               Define the Study Bbundaries
                Develop a Decision Rule   [^

                         *
             Specify Limits on Decision Errors
           Optimize the Design for Obtaining Data
                                                       IDENTIFY THE DECISION
           To define the decision statement that the
           study will attempt to resolve.

           Activities      '
           • Identify the principal study question.

           • Define the alternative actions that could
             result from resolution of the principal study
             question.

           • Combine the principal study question and the
             alternative actions into a decision statement.

           • Organize multiple decisions.
Purpose


       The purpose of this step is to define the decision statement that the study will attempt
to resolve.


Expected Outputs


       •  A decision statement that links the principal study question to possible actions
          that will solve the problem.
EPA QA/G-4
13
September 1994

-------
Background

       The goal of this step is to define the question that the study will attempt to resolve
and identify the alternative actions that may be taken based on the outcome of the study.  In
the DQO Process the combination of these two elements is called the decision statement or
decision.  The decision statement is critical for defining decision performance criteria later hi
the Process.

       The three activities in this chapter usually are most easily developed hi the order that
they appear.  Sometimes,  however, it is easier to identify alternative actions before the
principal study question.  In these cases, identify alternative actions that address the  problem,
then define the principal study question.

       In some cases, several decision statements are appropriate to address the problem
under investigation.  In these instances, the planning team should organize the decision
statements in order of priority and identify the most logical and efficient sequence for
analyzing and resolving them. If the principal study question is not obvious and specific
alterative actions cannot be identified, then the study may fall hi the category  of exploratory
research, in which case this step of the DQO  Process may not be applicable.

Activities

Identify the principal study question. Based on a review of the problem stated hi  Step  1,
identify the principal study question  and state it as specifically as possible.  A specific
statement of the principal  study question narrows the search for information needed to address
the problem.  The  principal study question identifies key unknown conditions  or unresolved
issues that reveal the solution to the  problem being investigated.  The following examples
illustrate this point:

       •  "Is the permittee out of compliance with discharge limits?"

       •  "Does the pollutant concentration exceed the National Ambient Air Quality
          Standard?"

       •  "Is the contaminant concentration significantly above background levels (which
          would indicate that a release has occurred)?"

Note that, hi each  case, the answer to the principal study question will provide the basis for
determining what course of action should be taken to solve the problem.
EPAQA/G-4                                   14         •                    September 1994

-------
Define the alternative actions that could result from resolution of. the principal study
question. Identify the possible actions that may be taken to solve the problem, including the
alternative that does not require action.  The types of actions considered will depend logically
on the possible answers to the principal study question.  These alternative actions form the
basis for defining decision performance criteria in Step 6: Specify Tolerable Limits on
Decision Errors.

       The following example illustrates how alternative actions are defined based on
possible answers to the following principal study question: "Are the lead pellets that are fired
by bird hunters and collect on the bottom of ponds contributing to the decrease in the duck  .
population in Adelayed County?" Possible resolutions of the principal study question are
1) the lead pellets are a factor in the decrease of the duck population, or 2) the lead pellets
are not a factor in the duck population's decrease.  If the lead is a contributing factor, the
action may be to remove the lead from the bottom of the ponds and, at the same time,
regulate the type of pellets that hunters may use in the future.  If lead pellets are not found to
contribute to a decrease in the duck population, then no action will be taken.

Combine the principal study question and the alternative actions into a decision
statement  Combine the alternative actions identified in the previous activity and the
principal study question into a decision statement that expresses a choice among alternative
actions. The following standard form may be helpful hi drafting decision statements:
"Determine whether or not [unknown environmental conditions/issues/criteria from the
principal study question] require (or support) [taking alternative actions]."

       To illustrate the decision statement framing activity, consider the previous example.
The principal study question is, "Are lead pellets on the bottom of ponds hi Adelayed County
contributing to  the decrease in the duck population?", and the alternative actions are to
"remediate the lead and regulate the use of lead pellets for hunting," or "take no action."
Therefore the decision statement is, "Determine whether or not lead pellets are contributing to
the decrease hi the duck population and require remediation and regulation." For a
compliance monitoring problem, a decision statement that incorporates the principal study
question and expresses a choice among alternative actions might be, "Determine whether or
not the permittee is out of compliance with discharge limits and requires enforcement action."

Organize multiple decisions. If several separate decision statements must be defined to
address the problem, list them and identify the sequence hi which they should be resolved. It
may be useful to document the decision resolution sequence and relationships in a diagram or
flowchart (see example in Figure 2-1).
EPAQA/G-4                                   15         '                    September  1994

-------
                                                                     	-(  Stop   J
Is contamination present?
                          Does contamination
                          pose unacceptable
                                risk?
             Determine extent of
          unacceptable contamination
                             Investigate possible remedies.
                                   Choose Remedy
                                    Apply remedy
                                   Is remedy working?
                                                                               Stop
            Final Goal Achieved?
        Figure 2-1.  Example of Multiple Decisions Organized Into a Flowchart.
EPA QA/G-4
                       16
September 1994

-------
                                     CHAPTER 3


           STEP 3:  IDENTIFY THE INPUTS TO THE DECISION
  THE DATA QUALITY OBJECTIVES PROCESS


State the Problem
*
Identifyttiepfldgion
^^^ \
Identify Inputs to the Decision
X^ \
Definelhe Study Boundaries
^s^
Develop a DecisionsRule
••^

* \

Specify Limits on Decision Errors

            Optimize the Design for Obtaining Data
                                                          IDENTIFY INPUTS

                                                      Purpose

                                                      To identify the informational inputs that will be
                                                      required to resolve the decision statement and
                                                      determine which inputs require environmental
                                                      measurements.   ,


                                                      Activities
                                                     • Identify the information that will be
                                                       required to resolve the decision statement


                                                     • Determine the sources for each item of
                                                       information identified.


                                                     • Identify the information that is needed
                                                       to establish the action level.


                                                     • Confirm that appropriate analytical
                                                       methods exist to provide the necessary
                                                       data.
Purpose


       The purpose of this step is to identify the informational inputs that will be required to

resolve the decision statement and determine which inputs require environmental
measurements.
                        \

Expected Outputs


       •  A list of informational inputs needed to resolve the decision statement.


       •  A list of environmental variables or characteristics that will be measured.
EPA QA/G-4
17
September  1994

-------
Background

       To resolve most decision statements, it is necessary to collect data or information.  In
this step, the planning team identifies the different types of information that will be needed to
resolve the decision statement.  The key information requirements include the measurements
that may be required, the source of data or information (e.g., historic or new data), and the
basis for setting the action level.  Once the planning team has determined what needs to be
measured, they will refine the specifications and criteria for these measurements' in later steps
of the DQO Process.

Activities

Identify the information that will be required to resolve the decision statement
Determine which environmental variables or other information are needed to resolve the
decision statement. Consider whether monitoring or modeling approaches, or a combination
of both, will be used to acquire the information. Based on the selected'data acquisition
approach, identify the types of information needed to support die decision statement. Ask
general questions such as, "Is information on the physical properties of the media required?"
or "Is information on the chemical characteristics of the matrix  needed?"  These types of
questions and their answers help identify the information needs. In compliance monitoring
for pollutants  discharged into surface water, examples of environmental variables of interest
may include levels of lead, silver, total suspended  solids, or temperature measurements.

Determine the sources for each item of information identified above.  Identify and list the
sources for the information needed to resolve the decision statement.  These sources may
include results of previous data collections, historical records, regulatory guidance,
professional judgement, scientific literature, or new data collections.  Next, qualitatively
evaluate whether any existing data are appropriate for the study. Existing data will be
evaluated quantitatively in Step 7:  Optimize the Design for Obtaining Data.
                                                                        t
Identify the information that is needed to establish the action level. Define the basis for
setting the action level.  The action level is me threshold value  which provides the criterion
for choosing between alternative actions. Action levels may be based on regulatory
thresholds or standards, or they may  be derived from problem-specific considerations such as
risk analysis.  In this  step, simply determine the criteria that will be used to set the numerical
value.  The  actual numerical action level will be set in Step 5: Develop a Decision Rule.

Confirm that appropriate measurement methods exist to provide the  necessary data.
Use the list  of environmental measurements identified earlier in this step  to develop a list of
potentially appropriate measurement  methods. Note the method detection limit and limit of
quantitation for each potential method; this performance information will be used in steps 5
and 7 of the DQO Process.
EPAQA/G-4                                   18         '                     September 1994

-------
                                     CHAPTER 4

           STEP 4:  DEFINE THE BOUNDARIES OF THE STUDY
 THE DATA QUALITY OBJECTIVES PROCESS



State the Problem
*
Identify the Decision ^S
*^
Identifylppdfsto the Decision
^ *
Define the Study Boundaries
\ *
Develop^Qecision Rule
* \
s


Specify Limits on Decision Error^" .

           Optimize the Design for Obtaining Data
                                                       DEFINE BOUNDARIES
                                                     Purpose
                                                      To define the spatial and temporal
                                                      boundaries that are covered by the
                                                      decision statement.
                                                     Activities        >
                                                     • Specify trie characteristics that define
                                                       the population of interest

                                                     • Define the geographic area
                                                       within which all decisions must apply.

                                                     • When appropriate, orvkte the population into
                                                       strata that have relatively homogeneous
                                                       characteristics.
                                                     • Determine the timefrarne to which the
                                                       decision applies.

                                                     • Determine when to collect data.
                                                     • Define the scale of decision making.
                                                     • Identify any practical constraints
                                                       on data collection.
Purpose

       The purpose of this step is to define the spatial and temporal boundaries of the
problem.

Expected Outputs

       *  A detailed description of the spatial and temporal boundaries of the problem.

       •  Any practical constraints that may interfere with the study.
EPA QA/G-4
19
September  1994

-------
Background

       It is difficult to interpret data that have not been drawn from a well-defined
population.  The term "population" refers to the total collection or universe of objects or
people to be studied, from which samples will be drawn.  The purpose of this step is to
define spatial and temporal components of the population that will be covered by the decision
statement so that the data can be easily interpreted.  These components include:

       •  Spatial boundaries that define the physical area to be studied and from where the
          samples should be taken, and

       •  Temporal boundaries  that describe the timeframe the study data will represent and
          when the samples should be taken.

       The boundaries will be used to ensure that the data collection design incorporates the
time periods in which the study should be implemented, areas that should be sampled, and the
time period to which the study results should apply. This will help ensure that the study data
are representative of the population being studied.  Defining boundaries before the data are
collected can also prevent inappropriate pooling of data hi a way that masks useful
information.

       Practical constraints that could interfere with sampling should also be identified in this
step.  A practical constraint is any  hinderance or obstacle that potentially may interfere with
the full implementation of the data collection design.

Activities

Specify the  characteristics that define the population of interest  Specify the
characteristics that define the population.  It is important to clearly define the attributes that
make up the population by  stating them in a way that makes the focus of the study
unambiguous. For example, the  population may be PCB concentrations in soil, lead
concentrations in the blood of children under the age of seven, or hourly ozone concentrations
within the metropolitan area. There may be  several ways to define a population; always
choose the one that is most specific.  For example, "tetrachlorodibenzodioxin" is more
specific than "dioxin,"  and "hexavalent chromium"  is more specific than "chromium".

Define the spatial boundary of the decision statement

       Define the geographic area to which the decision statement applies. The
       geographic area is a region distinctively marked by some physical features (i.e.,
       volume, length, width, boundary). Some examples of geographic areas are the
       metropolitan city  limits, the soil within the property boundaries down to a depth of six
       inches, or the natural habitat range of a particular animal species.
EPAQA/G-4                                   20         •                    September 1994

-------
       When appropriate, divide the population into strata that have relatively
       homogeneous characteristics. Using existing information, stratify or segregate the
       elements of the population into subsets or categories that exhibit relatively
       homogeneous properties or characteristics that may have an'influence on the outcome
       of the study, such as contaminant concentrations, age, or height. Dividing the
       population into strata is desirable for studying sub-populations, reducing variability
       within subsets of data, or reducing the complexity of the problem by breaking it into
       more manageable pieces.  See Figure 4-1 for an example of how to stratify a site with
       soil contamination.

Define the temporal boundary of the problem.

       Determine the timeframe to which the decision applies.  It may not be possible to
       collect data over the full tune period to which the decision will apply.  Therefore the
       planning team should determine the timeframe that the data should reflect; for
       example, "The data will reflect the condition of contaminant leaching into ground
       water over a period of a hundred years,"  or "The data will be used to reflect the risk
       conditions  of an average resident over their average length of residence which is
       estimated to be eight years."  Timeframes should be defined for the overall population
       and any sub-populations of interest.

       Determine when to collect data.  Conditions may vary over the course of a study,
       which may affect the success of data collection  and the interpretation of data results.
       These factors may include weather, temperature, humidity, or amount of sunlight and
       wind.  Determine when conditions will be most favorable for collecting data and select
       the most appropriate time period to collect data that reflect those conditions. For
       example, a study to measure ambient airborne paniculate matter may give misleading
       information if the sampling is conducted in the wetter winter months rather than the
       drier summer months.

Define the scale of decision making.  Define the smallest, most appropriate subsets of the
population (sub-populations) for which decisions will be made based  on the spatial or
temporal boundaries. For example, in a study where the decision statement is, "Determine
whether or not the concentration of lead in soil poses an unacceptable health risk to children
and requires remediation", the geographic area is the top six inches of soil within the
property boundaries, and the population is the lead concentration in surface soil.  The scale of
decision making could be set to an area which has a size that corresponds to the area where
children derive the majority of their exposure (such as a play area or an average residential
lot size if the future land use will be residential).  Studying the site at this scale will be
protective of children, a sensitive population in risk assessment.  A temporal scale of decision
making might be necessary for other types of studies. For example, in order to regulate water
quality, it would be useful to set a scale of decision majcing that limits the time between
sampling events.  This would minimize the potential adverse effects in case the water quality
was degraded between sampling events.

EPAQA/G-4                                   21         '                    September 1994

-------
Identify any practical constraints on data collection.  Identify any constraints or obstacles
that could potentially interfere with the full implementation of die data collection design, such
as seasonal or meteorological conditions when sampling is not possible, the inability to gain
site access or informed consent, or the unavailability of personnel, 'time, or equipment. For
example, it may not be possible to take surface soil samples beyond the east boundaries of a
site under investigation because permission had not been granted by the owner of the adjacent
property.
                Stratification
Forested
Area
                                  Drum
                                  Disposal
                                  Area
                                  Possible
                      Main         de-watering
                      Building       treatment
                      and Grounds   area.
Forested
Area

(Stratum 1)

Main
Building
and Grounds
(Stratum 3)
Drum
Disposal
Area
(Stratum 2)
Possible
de-watering
treatment
area.
(Stratum 4)
                           Site A

                   Site stratification based on current and past land use.
                              Large stained area
                              w/pungent odor.
                      Visibly rusted
                      55 gallon
                      drums.
                                       Large stained area
                                       i w/pungent odor.
                                                     (Stratum 3)
                                             Stratum 2)
                            SiteB

                    Site stratification based on site inspection or preliminary
                    data.
       Figure 4-1.  An Example of How to Stratify a Site with Soil Contamination.
EPA QA/G-4
                      22
September 1994

-------
                                     CHAPTER 5

                    STEP 5:  DEVELOP A DECISION RULE
  THE DATA QUALITY OBJECTIVES PROCESS

State the Problem
*
Identify the Decision
*
Identify Inputs to the Decision
*^
Define Jhe"Study Boundaries
^ *
Develop a Decision Rule
^-^*
Specify Limits on De^toteoJErrors
/
/


     Optimize the Design for Obtaining Data
                                                     DEVELOP A DECISION RULE

                                                  Purpose

                                                   To define the parameter of interest,
                                                   specify the action level, and integrate previous
                                                   DQO outputs into a single statement that
                                                   describes a logical basis for choosing among
                                                   alternative actions.   ,

                                                  Activities

                                                   • Specify the statistical parameter that
                                                     characterizes the population.

                                                   • Specify the action level for the study.

                                                   • Combine the outputs of the previous DQO
                                                     steps into an "if—then...11 decision rule
                                                     that defines the conditions that would
                                                     cause the decision maker to choose
                                                     among alternative actions.
Purpose


       The purpose of this step is to define the parameter of interest, specify the action level,
and integrate previous DQO outputs into a single statement that describes a logical basis for
choosing among alternative actions.

Expected Outputs


       •  The statistical parameter (the parameter of interest) that characterizes the
          population.


       •  The action level.


       •  An "if...then..." statement that defines the conditions that would cause the
          decision maker to choose among alternative actions.
EPA QA/G-4
23
September  1994

-------
Background

       The decision.rule summarizes what attributes the decision maker wants to know about
the population and how that knowledge would guide the selection of a course of action to
solve the problem. The Decision Rule step combines criteria from past steps with the
parameter of interest (statistical characteristic of the population) and the action level to
provide a concise description of what action will be taken based on the results of the data
collection.

There are four main elements to a decision rule:

       (1)    The parameter of interest, a descriptive measure (such as a mean, median, or
              proportion) that specifies the characteristic or attribute that the decision maker
              would  like to know about the statistical population. The purpose of the data
              collection design is to produce environmental data that can be used to develop
              a reasonable estimate of the population parameter.

       (2)    The scale of decision making, the smallest,  most appropriate subset (sub-
              population) for which separate decisions will be made. (The scale of decision
              making was defined hi Step 4:  Define the Boundaries of the Study.)

       (3)    The action level, a measurement threshold value of the parameter of interest
              that provides the criterion for choosing among alternative actions.  The action
              level can be based on regulatory standards,  an exposure assessment, technology
              based limits, or reference-based standards.

       (4)    The alternative actions, the actions that the decision maker would take,
              depending on the true value of the parameter of interest.  (The alternative
              actions were identified hi Step 2:  Identify the Decision.)

Activities

Specify the statistical parameter that characterizes the  population (the  parameter of
interest).   The planning team should specify the parameter of interest (such as the mean,
median, or percentile) whose true value the decision maker would like know and that the data
will estimate.  For example, to determine if the contamination level at a given site exceeds an
action level, the planning team must specify the parameter that will be evaluated with respect
to the action level (e.g., the mean concentration). Some regulations specify the parameter, but
if this is not the case, it may be necessary to consult with a statistician to help select a
parameter that is consistent with the intended  application.  Recognize that the parameter that
is chosen  in this step may be changed to an equivalent descriptive measure as more
information becomes  available based on statistical considerations in Step 7  of the DQO
Process and in the Data Quality Assessment Process.  Information about positive and negative
attributes of commonly used parameters is provided at the end of this chapter.

EPAQA/G-4                                    24        '                    September  1994

-------
Specify the action level for the study.  The decision maker should specify the numerical
value that would cause him/her to choose between alternative actions.  For example, the
decision maker would choose one action if the true value of the parameter of interest is above
1 mg/L, and a different action otherwise. Confirm that the action level is greater than the
detection and quantitation limits for the potential measurement methods identified in Step 3:
Identify the Inputs to the Decision.

Develop a decision rule. Develop a decision rule as an "if...then..." statement that
incorporates the parameter of interest, the scale of decision making, the action level, and the
action(s) that would result from resolution of the decision.  These four elements are combined
in the following way:  If the parameter of interest (e.g., true mean concentration of lead)
within the scale of decision making (e.g., 1-acre plots) is greater than the action level
(e.g., 1  mg/Kg), then take alternative action A  (e.g., remove the soil from the site); otherwise
take alternative action B (e.g., leave the soil in place). For example, "If the true mean
concentration of cadmium in the fly ash  leachate within a container truck exceeds  1.0 mg/Kg,
then the waste ash will be considered hazardous and will be disposed of in a RCRA
hazardous waste landfill; otherwise, the waste ash will be disposed  of hi a municipal landfill."
This statement is a functional decision rule that expresses what the  decision maker ideally
would like to resolve.  It is not an operational decision rule  which incorporates the  decision
maker's  tolerable limits on decision errors and the statistical hypothesis, and describes how
the data will be  summarized.  The operational decision rule  is developed during  the Data
Quality Assessment Process, after the data have been collected (see Appendix A).
                  Attributes of Different Statistical Parameters

   MEAN

          Positive Attributes

          •  Useful when action level is based on long-term, average health effects
             (chronic conditions, carcinogenicity).
          •  Useful when the population is uniform with relatively small spread.
          •  Generally requires fewer samples than other parameters.

          Negative Attributes

          •  Not a very representative measure of central tendency for highly skewed
             populations.
          •  Not useful when the population contains a large proportion of values that are
             less than measurement detection limits.                          (continued)
               Table 5-1. Attributes of Different Statistical Parameters to
                               Characterize the Population

EPAQA/G-4                                   25          •                    September 1994

-------
             Attributes of Different Statistical Parameters (continued)
     MEDIAN
             Positive Attributes

             •   Useful when action level is based on long-term, average health effects (chronic
                conditions, carcinogenicity).
             •   Provides a more representative measure of central tendency than the mean for
                skewed populations.
             •   Useful when the population contains a large number of values that are less
                than measurement detection limits.
             •   Relies on few statistical assumptions.

             Negative Attributes
                Will not protect against the effect of extreme values.
                Not a very representative measure of central tendency for highly skewed
                populations.
     UPPER PROPORTION/PERCENTILE

             Positive Attributes

             •  Useful for protection against extreme health effects.
             •  For highly variable populations, provides best control of the extreme values.
             •  Useful for skewed distributions.
             •  May be appropriate when the population contains a large number of values
                less than the measurement detection limit, as long as this limit is less than the
                action level.
             •  Relies on few statistical assumptions.

             Negative Attributes

             •  Requires larger sample sizes than mean.

     Reference: U.S. Environmental Protection Agency. 1989. Methods for Evaluation Attainment of Cleanup Standards:
     Volume I: Soils and Solid Media. EPA 230/02-89-042, Office of Policy Planning and Evaluation.
           Table 5-1.  (cont)  Attributes of Different Statistical Parameters to
                                Characterize the Population
EPA QA/G-4
26
September 1994

-------
                                    CHAPTER 6

   STEP 6:  SPECIFY TOLERABLE LIMITS ON DECISION ERRORS
 THE DATA QUALITY OBJECTIVES PROCESS
/

State the Problem
*
Identify the Decision /
* /
Identify Inputs to thaOecision
V
Define the^tudy Boundaries
/ *
Jfevelop a Decision Rule
/ *
/
/
1 Specify Limits on Decision Errors 1
          Optimize the Design for Obtaining Data
                                                         SPECIFY LIMITS
                                                      ON DECISION ERRORS

                                                   Purpose
                                                    To specify the decision maker's tolerable limits
                                                    on decision errors.
                                                   Activities

                                                   •  Determine the possible range of the
                                                     parameter of interest.

                                                   •  Identify the decision errors and choose the
                                                     null hypothesis.
                                                     Specify a range of possible parameter values
                                                     where the consequences of decision errors
                                                     are relatively minor (gray region).
                                                     Assign probability values to points above and
                                                     below the action level that reflect the
                                                     tolerable probability for the
                                                     occurrence of decision errors.
Purpose

       The purpose of this step is to specify the decision maker's tolerable limits on decision
errors, which are used to establish performance goals for the data collection design.

Expected Outputs

       •  The decision maker's tolerable decision error rates based on a consideration
          of the consequences of making an incorrect decision.
EPA QA/G-4
27
September 1994

-------
Background

       Decision makers are interested in knowing the true state of some feature of the
environment.  Since data can only estimate this state, decisions that are based on
measurement data could be in error (decision error). Most of the time the correct decision
will be made; however, this chapter will focus on controlling the less likely possibility of
making a decision error. The goal of the planning team is to develop a data collection design
that reduces the chance of making a decision error to a tolerable level.  This step of the DQO
Process will provide a mechanism for allowing the decision maker to define tolerable limits
on the probability of making a decision error.

       There are two reasons why the decision maker cannot know the true value of a
population parameter (i.e.,  the true state of some  feature of the environment):

       (1)    The population of interest almost always varies over time and space.  Limited
              sampling will miss some features of this  natural variatiorl because it is usually
              impossible or impractical to measure every point of a population. Sampling
              design error occurs when the sampling design is unable to capture the
              complete extent  of natural variability that exists in the true state of the
              environment.

       (2)    Analytical methods and instruments are never absolutely perfect, hence a
              measurement can only estimate the true value of an environmental sample.
              Measurement error refers to a combination of random and systematic errors
              that inevitably arise during the various steps of the measurement process (for
              example, sample collection, sample handling, sample preparation, sample
              analysis, data reduction, and data handling).

       The combination of sampling  design error and measurement error is  called total study
error, which may lead to a decision error. Since it is impossible to eliminate error in
measurement data, basing decisions on measurement data will lead to the possibility of
making a decision error.

       The probability of decision errors can be controlled by adopting a scientific approach.
In this approach, the data are used to select between one condition of the environment (the
null hypothesis, HJ and an alternative condition (the alternative hypothesis,  HJ.  The null
hypothesis is treated like a baseline condition that is presumed to be true hi the absence of
strong evidence to the contrary. This feature provides a way to guard against making the
decision error that the decision maker considers to have the more undesirable consequences.

       A decision error occurs when  the decision maker rejects the null hypothesis when it is
true, or fails to reject the null hypothesis when it is false. These two types  of decision errors
are classified as false positive and false negative  decision errors, respectively. They are
described below.

EPAQA/G-4                                   28          '                    September 1994

-------
        False Positive Decision Error — A false positive decision error occurs when the null
hypothesis (HJ is rejected when it is true.  Consider an example where the decision maker
presumes that a certain waste is hazardous (i.e., the null hypothesis or baseline condition is
"the waste is hazardous").  If the decision maker concludes that there is insufficient evidence
to classify the waste as hazardous when it truly is hazardous, then the decision maker would
make a false positive decision error.  A statistician usually refers to the false positive error as
a "Type I" error.  The  measure of the size of this error is called alpha (a), the level of
significance, or the size of the critical region.

        False Negative Decision Error — A false negative decision error occurs when the
null hypothesis is not rejected when it is false.  In the above waste example, the false
negative decision error occurs when the decision maker concludes that the waste is hazardous
when it truly is not hazardous. A statistician usually refers to a false negative error as a
"Type II" error.  The measure of the size of this error is called beta (|3),  and is also known as
the complement of the power of a hypothesis test.
                                                                     i
        The definition of false positive and false negative decision errors  depends on the
viewpoint of the decision maker.1 Consider the viewpoint where a person has been presumed
to be "innocent until proven guilty" (i.e., H0 is "innocent"; H, is "guilty").  A false positive
error would  be convicting an  innocent person; a false negative error would be not convicting
the guilty person.  From the viewpoint where a person is presumed to be "guilty until proven
innocent" (i.e., H0 is "guilty"; H, is "innocent"), the errors are reversed. Here, the false
positive error would be not convicting the guilty person, and the false negative error would be
convicting the innocent person.

        While the  possibility of a decision error can  never be totally eliminated, it can be
controlled.  To control the  possibility of making decision errors, the planning  team must
control total study error.  There are many ways to accomplish this, including collecting a
large number of samples (to control sampling design error), analyzing individual samples
several times or using more precise laboratory methods (to control measurement error).
Better sampling designs can also be developed to  collect data that more accurately and
efficiently represent the population of interest. Every study will use a slightly different
method of controlling decision errors, depending on where the largest components of total
study error exist in the data set and the ease of reducing those error components. Reducing
the probability of making decision errors generally increases costs. In many cases controlling
decision error within very small limits is unnecessary for making a decision that satisfies the
decision maker's needs.  For instance, if the consequences of decision errors are minor, a
reasonable decision could be made based on relatively crude data (data with high total study
   'Note that these definitions are not the same as false positive or false negative instrument readings, where
similar terms are commonly used by laboratory or field personnel to describe a fault in a single result; false
positive and false negative decision errors are defined in the context of hypothesis testing, where the terms are
defined with respect to the null hypothesis.

EPAQA/G-4                                    29        .                      September 1994

-------
error). On the other hand, if the consequences of decision errors are severe, the decision
maker will want to control sampling design and measurement errors within very small limits.

       To minimize unnecessary effort controlling decision errors, 'the planning team must
determine whether reducing sampling design and measurement errors is necessary to meet the
decision maker's needs.  These needs are made explicit when the decision maker specifies
probabilities of decision errors that are tolerable. Once these tolerable  limits on decision
errors are defined, then the effort necessary to analyze and reduce sampling design and
measurement errors to satisfy  these limits can be determined in Step 7:  Optimize the  Design
for Obtaining  Data.  It may be necessary to iterate between these two steps before finding
tolerable probabilities of decision errors that are feasible given resource constraints.

Activities

Determine the possible range of the parameter of interest  Establish the possible range of
the parameter of interest by estimating its likely upper and lower bounds.  This will help
focus the remaining activities  of this step on only the relevant values of the parameter. For
example, the range of the parameter shown hi Figures 6-1 and 6-2 at the end of this chapter
is between 50 and 200 ppm. Historical and documented analytical data are of great help in
establishing the potential parameter range.

Identify the decision errors and choose the null hypothesis.  Define where each decision
error occurs relative to the action level and establish which decision error should be defined
as the null hypothesis (baseline condition).   This process has four steps:

       (1)     Define both types of decision errors and establish the true state of nature for
              each decision error.  Define both types of decision errors and determine which
              one occurs above and which one occurs below the action level.  A decision
              error occurs when the data mislead the decision maker into concluding that the
              parameter of interest is on one side of the action level when the true value of
              the parameter is on the other side of the action level. For example, consider a
              situation in which a study is being conducted to determine if mercury
              contamination is creating a health hazard and EPA wants to take action if more
              than 5% of a population of fish have mercury levels above a risk-based action
              level. In this case, a decision error would occur if the data lead the decision
              maker to conclude that 95%  of the mercury levels found in the fish population
              were below the action level (i.e., the parameter is the "95th percentile" of
              mercury levels hi the fish population) when the true 95th percentile of mercury
              levels in the fish population  was above the action level (which means that  more
              than 5% of the  fish population contain mercury levels greater than the action
              level). The other decision error  for this example would be that the data lead
              the decision maker to conclude that the 95th percentile of mercury levels in the
              fish population  is greater than the action level when the true 95th percentile is
              less than the .action level.

EPAQA/G-4                                  30        •                    September 1994

-------
              The "true state of nature" is the actual condition or feature of the environment
              that exists, but is unknown to the decision maker.  Each decision error consists
              of two parts, the true state of nature and the conclusion that the decision maker
              draws.  Using the example above, the true state of nature for the first decision
              error is that the 95th percentile of mercury levels in the fish population is
              above the action level.

       (2)    Specify and evaluate the potential consequences of each decision  error.
              Specify the likely consequences of making each decision error and evaluate
              their potential severity in terms of economic and social costs, human health and
              ecological effects, political and legal ramifications, and so on.  Consider the
              alternative actions that would be taken under each decision error scenario, as
              well as secondary effects of those actions.  For example, in determining
              whether or not 95% of a fish population contain mercury levels above a risk-
              based action level, there may be a variety of potential consequences of
              committing a decision error.  In the first decision error described above, where
              the decision maker  concludes that the 95th percentile is below when the true
              95th percentile was above the action level,  the decision maker may decide to
              continue to allow fishing in the waters and not undertake any cleanup activity.
              The resulting consequences might include human health and ecological effects
              from consumption of contaminated fish by humans and other animals,
              economic and social costs of health care and family disruption, and damaged
              credibility of EPA when (and if) the decision error is detected.  If the other
              type of decision error is committed, where the decision  maker decides that the
              95th percentile exceeds the action level when the true 95th percentile is below
              the action level, the decision maker might ban all fishing in the local waters
              and initiate cleanup activities. The consequences might include economic and
              social costs of lost revenues and job displacement in the fishing industry,
              damaged credibility for EPA when the cleanup activities expose the nature of
              the decision error, and the threat of lawsuits by fishing interests.

              Evaluate the severity of potential consequences of decision errors  at different
              points within the domains of each type of decision error, since the severity  of
              consequences may change as the parameter moves further away from the action
              level. Consider whether or not the consequences change abruptly at some
              value, such as a threshold health effect level; the decision maker may want to
              change the tolerable limit on the decision error at such a point.

       (3)    Establish which decision error has more severe consequences near the action
              level.  Based on the evaluation of potential  consequences of decision errors, the
              decision maker should determine which decision error causes greater concern
              when the true parameter value is near the action level.  It is important to focus
              on the region near the action level because this is where the true parameter
              value is most .likely to be when a decision error is made (in other words, when

EPAQA/G-4                                   31          '                    September 1994

-------
              the true parameter is far above or far below the action level, the data are much
              more likely to indicate the correct decision).  This determination typically
              involves value judgements about the relative severity of different types of
              consequences within the context of the problem. In the fish contamination
              problem above, the decision maker would weigh the potential health
              consequences from allowing people to consume contaminated fish versus the
              economic and social disruption from banning all fishing in the community.  In
              this case, the decision maker might carefully consider how uncertain or
              conservative the risk-based action level is.

       (4)    Define the null hypothesis (baseline condition) and the  alternative hypothesis
              and assign the terms  "false positive" and "false negative" to the appropriate
              decision error.  In problems that concern regulatory compliance, human health,
              or ecological risk, the decision error  that has the most adverse potential
              consequences should be  defined as the null hypothesis (baseline condition).2
              In statistical hypothesis testing, the data must conclusively demonstrate that the
              null hypothesis is false.  That is, the data must provide enough information to
              authoritatively reject the null hypothesis (disprove the baseline condition) in
              favor of the alternative.  Therefore, by setting the null hypothesis equal to the
              true state of nature that exists when the more severe decision error occurs, the
              decision maker guards against making the more severe  decision error by
              placing the burden of proof on demonstrating that the most adverse
              consequences will not be likely to occur.

              It should be noted that the null and alternative hypotheses have been
              predetermined in many regulations.   If not, the planning team should define the
              null hypothesis (baseline condition) to correspond to the true state  of nature for
              the more severe decision error and define the alternative hypothesis to
              correspond to the true state of nature for the less severe decision error.

              Using the definitions of  null and alternative hypotheses, assign the term "false
              positive" to the decision error hi which the decision maker rejects the null
              hypothesis when it is true, which corresponds to the decision error with the
              more severe consequences identified in task (3). Assign the term "false
              negative" to the decision error in which the decision maker fails to reject the
   2Note that this differs somewhat from the conventional use of hypothesis testing in the context of planned
experiments. There, the alternative hypothesis usually corresponds to what the experimenter hopes to prove, and
the null hypothesis usually corresponds to some baseline condition that represents an "opposite" assumption. For
instance, the experimenter may wish to prove that a new water treatment method works better than an existing
accepted method.  The experimenter might formulate the null hypothesis to correspond to "the new method
performs no better than the accepted method," and the alternative hypothesis as "die new method performs better
than the accepted method." The burden of proof would then be on the experimental data to show that the new
method performs better than the accepted method, and that this result is not due to chance.

EPAQA/G-4                                    32         '                     September 1994

-------
              null hypothesis when it is false, which corresponds to the decision error with
              the less severe consequences identified in task (3).

Specify a range of possible parameter values where  the consequences of decision errors
are relatively minor (gray region). The gray region is a range of possible parameter values
where the consequences of a false negative decision error are relatively minor.  The gray
region is bounded on one side by the action level and on the other side by that parameter
value where the consequences  of making a false negative  decision error begin to be
significant.  Establish this boundary by evaluating the consequences of not rejecting the null
hypothesis when it is false.  The edge of the gray region should be  placed where these
consequences are severe enough to set a limit on the magnitude of this false negative decision
error.  Thus, the gray region is the area between this parameter value and the action level.

       It is necessary to specify a gray region because variability in the population and
unavoidable imprecision in the measurement system combine to produce variability in the
data such that a decision may be "too close to call" when the true parameter value is  very
near the action level. Thus,  the gray region (or "area of uncertainty") establishes the
minimum distance from the action level where the decision maker would like to begin to
control false negative decision errors. In statistics, the  width of this interval is called the
"minimum detectable difference" and is often expressed as the Greek letter delta (A).  The
width of the gray region is an essential part of the calculations for determining the number of
samples needed to  satisfy the DQOs, and represents one important aspect of the  decision
maker's concern for decision errors.  A more narrow gray region implies a desire to detect
conclusively the condition when the true parameter value is close to the action level ("close"
relative to the variability in the data). When the true value of the parameter falls within the
gray region, the decision maker may face a high probability of making a false negative
decision error, since the data may not provide conclusive evidence for rejecting the null
hypothesis,  even though it is actually false (i.e., the data may be too variable to allow the
decision maker to recognize  that the presumed baseline condition is, in fact, not  true).

       From a practical standpoint, the gray region is an area where it will not be feasible or
reasonable to  control the false negative decision error rate to low levels because  of high costs.
Given the resources that would be required to reliably detect small differences between the
action level and the true parameter value, the decision maker must balance the resources spent
on data collection with the expected consequences of making that decision error.  For
example, when testing whether a parameter (such as the mean concentration) exceeds  the
action level, if the true  parameter is near the action level (relative to the expected variability
of the data), then the imperfect data will tend to be clustered around the action level,  with
some values above the  action level and some below.  In this situation, the likelihood of
committing  a  false negative decision error will be large. To determine with confidence
whether the true value of the parameter is above or below the action level, the decision maker
would need to collect a large amount of data, increase the precision of the measurements,  or
both.  If taken to an extreme, the cost of collecting data can exceed the cost of making a
decision error, especially where the consequences of the decision error may be relatively

EPAQA/G-4                                    33         '                    September 1994

-------
minor.  Therefore, the decision maker should establish the gray region, or the region where it
is not critical to control the false negative decision error, by balancing the resources needed to
"make a close call" versus the consequences of making that decision error.

Assign probability limits to points above and below the gray region that reflect the
tolerable probability for the occurrence of decision errors. Assign probability values to
points above and below the gray region that reflect the decision maker's tolerable limits for
making an incorrect decision.  Select a possible value of the parameter; then choose a
probability limit based on an evaluation of the seriousness of the potential consequences of
making the decision error if the true parameter value is located at that point.  At a minimum,
the decision maker should specify a false positive decision error limit at the action level, and
a false negative decision  error limit at the other end of the gray region.  For many situations,
the decision maker may wish to specify additional probability limits at other possible
parameter values.  For example, consider a hypothetical toxic substance that has a regulatory
action level of 10 ppm, and which produces threshold effects in humans exposed to mean
concentrations above 100 ppm. In this situation, the decision maker may wish to specify
more stringent probability limits at that threshold concentration of 100 ppm than those
specified at 10 ppm.  The tolerable decision error limits should decrease further away from
the action level as the consequences of decision error become more severe.

       Given the potentially high cost of controlling sampling design  error and measurement
error for environmental data, Agency decision making is rarely supported by  decision error
limits more stringent than 0.01 (1%) for both the false positive and false negative decision
errors.  This guidance recommends using 0.01 as the starting point for setting decision error
rates.  The most frequent reasons for setting limits greater (i.e., less stringent) than 0.01 are
that the consequences of  the decision errors may not be severe enough to warrant setting
decision error rates that are this extreme.  The value of 0.01 should not be considered a
prescriptive value for setting decision error rates, nor should it be considered as the policy of
EPA to encourage the use of any particular decision error rate.  Rather, it should be viewed
as a starting point from which to develop limits on decision errors that are applicable  for each
study. If the decision maker chooses to relax the decision error rates  from 0.01 for false
positive or false negative decision errors,  the planning team should document the reasoning
behind setting the less stringent decision error rate and the potential impacts on cost, resource
expenditure, human health, and ecological conditions.

       The combined information from the activities section of this chapter can be graphed
onto a "Decision Performance Goal Diagram" or charted in a "Decision Error Limits Table"
(see Figures 6-1 and  6-2  and Tables 6-1 and 6-2 below). Both are useful tools for visualizing
and evaluating all of the outputs from this step. Figure 6-1 and Table 6-1 illustrate the case
where the null hypothesis (baseline condition) is that the parameter of interest exceeds the
action level (e.g., the waste is hazardous). Figure 6-2 and Table 6-2 illustrate the case where
the null hypothesis (baseline condition) is that the parameter is less than the action level (e.g.,
the waste is not hazardous).
EPAQA/G^                                   34         •                     September 1994

-------
        0>
       £

       73
       I

       1
       Q
    1.0


    0.9


;    o.s



!    °7

I   .0.6


:    0.5


|    0.4


i    0.3
I

j    0.2


    0.1
 0.05 —
                       Tolerable
                        False
                       Negative

                      Error Rates

                                                          Tolerable
                                                            False
                                                           Decision
                                                          Error Rates
Gray Region

(Relatively Urge
Decision Error
   Rates are
  Considered
  Tolerable.)
                                                                              1.0
                                                                                 0.99
0.95

0.9


0.8


0.7


0.6


0.5


0.4


0.3


0.2


0.1
                      50  I  70   I  90  I  110  I  130  I 150  !  170  I  190 I
                         60     80    100    120   140    160   180    200

                                       t— Action Level

                   True Value of the Parameter (Mean Concentration, ppm)


            Figure 6-1. An Example of a Decision Performance Goal Diagram

                   Baseline Condition: Parameter Exceeds Action Level.
True
Concentration
< 60 ppm
60 to 80
80 to 100
100 to 150
> 150
Correct
Decision
Not exceed
Not exceed
Not exceed
Does exceed
Does exceed
Type of
Error
F(-)
F(-)
F(-)
F(+)
F(+)
Tolerable Probability of
Incorrect Decision
5%
10%
gray region
5%
1%
          Table 6-1. Decision Error Limits Table Corresponding to Figure 6-1.

                                 (Action Level = 100 ppm)
EPA QA/G-4
                                 35
                              September 1994

-------
          
         i
         8
         Q
                  1.0


                  0.9
             I   0.8
.2

S
0>
            2
a. 2
   CO
   0.
   0.7


   0.6





   0.4


   0.3


   0.2


   0.1

0.05  -
                            Tolerable
                             False
                             Positive
                           Error
                               \
                        50
                  70  1  90
110
                                                              Tolerable
                                                                False
                                                              Error
                                             Gray Region

                                              (Relatively Large
                                              Decision Etnx
                                                           Tolerable.)
                                         130 I 150170  I 190  I
                                                                   1.0

                                                                   —0.95
                                    0.9


                                    0.8


                                    0.7


                                    0.6


                                    0.5


                                    0.4


                                    0.3


                                    0.2


                                    0.1


                                      0
                                        i ww
                                             Action Level

                     True Value of the Parameter (Mean Concentration, ppm)


             Figure 6-2.  An Example of a Decision Performance Goal Diagram

                 Baseline Condition:  Parameter Is Less Than Action Level.
True
Concentration
< 60 ppm
60 to 100
100 to 120
120 to 150
> 150
Correct
Decision
Not exceed
Not exceed
Does exceed
Does exceed
Does exceed
Type of
Error
F(+)
F(+)
F(-)
F(-)
F(-)
Tolerable Probability of
Incorrect Decision
5%
10%
gray region
20%
5%
          Table 6-2.  Decision Error Limits Table Corresponding to Figure 6-2.
EPA QA/G-4.
                                36
                                  September  1994

-------
                               CHAPTER?
      STEP 7: OPTIMIZE THE DESIGN FOR OBTAINING DATA
THE DAT
•
Z
A QUALITY OBJECTIVES PROCESS




State the Problem
*
Identify the Decision >
* /
Identify Inputs to the Decision
* /
Define the Study Boundaries
A
Develop a Decision Rule
/ *
Specify Limits on Decision Errors
/

/ \\^~~*
/
/
•7
OPTIMIZE THE DESIGN
Purpose
cotodtan design tor generating
data that am expected to satisfy the OQOs.
Activities
• Review the DQO outputs and existing
envlrofwnentai data.
• Develop general data collection
* Formulate the mathematical expressions
needed to solve the design problems

• Select the optimal sample size that

satisfies all of the OQOs.
• Document the operational details and
theoretical assumptions of the selected
design in the sampling and analysis plan.
^-—

r ^^~~~ \ 	
I Optifliua lllll III sign for Obtaining Data 1 ^ -""^





Purpose

      The purpose of this step is to identify a resource-effective data collection design for
generating data that are expected to satisfy die DQOs.

Expected Outputs

      •  The most resource-effective design for the study that is expected to achieve
         the DQOs.
EPA QA/G-4
37
                          September 1994

-------
Background

       In this step, statistical techniques are used to develop alternative data collection
designs and evaluate their efficiency in meeting the DQOs. To develop the optimal design
for this study, it may be necessary to work through this step more than once after revisiting
previous steps of the DQO Process.

       The objective of this step is to identify the most resource-effective data collection
design expected to generate data that satisfy the DQOs specified in the preceding steps.
While a full explanation of the procedures for developing a data collection design is beyond
the scope of this guidance document, it does provide a broad overview of the steps that need
to be accomplished to reach this goal. The example in Appendix B illustrates some of these
activities in more detail.

Activities
                                                                  j
Review the DQO outputs and existing environmental data. Review the DQO outputs
generated in the preceding  six steps to ensure that they are internally consistent.  The DQOs
should provide a succinct collection of information on the context of, requirements for, and
constraints on the data collection design. Review existing data in more detail if it  appears
that they can be used to support the data collection design (e.g.,  analyze the variability in
existing data if they appear to provide good information about the variance for the new data).
If existing data are going to be combined with new data to support the decision, then
determine if there are any gaps that can be filled or deficiencies  that might be mitigated by
including appropriate features in the new data collection design.

Develop general data collection design alternatives.  Develop alternative data collection and
analysis designs based on the DQO outputs and other relevant information, such as historical
patterns of contaminant deposition, estimates of variance, and technical characteristics of the
contaminants and media. Generally, the goal is to find cost-effective alternatives that balance
sample size and measurement performance, given the feasible choices for sample collection
techniques and analytical methods.  In some cases where there is a relatively high spatial or
temporal variability, it may be more cost-effective to use less expensive yet less  precise
analytical methods so that a relatively large number of samples can be taken,  thereby
controlling the sampling design error component of total study error. In other cases where
the contaminant distribution is relatively homogeneous, or the action level is very near the
method detection limit, it may be more cost-effective to use more expensive yet  more precise
and/or more sensitive analytical methods and collect fewer samples, thereby controlling the
analytical measurement error component of total study error.  Examples of general data
collection design alternatives include:

       •   factorial design                        •  sequential random sampling
       •   simple random sampling               •  systematic sampling
       •   stratified random sampling            •  composite sampling (in conjunction
                                                   with another sampling design)


EPAQA/G-4                                  38                              September 1994

-------
Formulate the mathematical expressions needed to solve the design problem for each
data collection design alternative. Develop the following three mathematical expressions
needed to optimize the data collection design as follows:

        (1)     Define a suggested method for testing the statistical hypothesis and define a
               sample size formula that corresponds to the method if one exists
               (e.g., a Student's t-test).

        (2)     Develop a statistical model that describes the relationship of the  measured
               value to the "true" value.  Often the model will describe the components of
               error or bias that are believed to exist hi the measured value.

        (3)     Develop a cost function that relates the number of samples to the total cost of
               sampling and analysis.

Select the optimal sample size that satisfies the DQOs for each data* collection design
alternative. Using the mathematical expressions from the previous activity, solve for the
optimal sample size that satisfies the DQOs, including the decision maker's limits on decision
errors.  If no design will  meet the limits on decision errors within  the budget or other
constraints, then the planning team will need to relax one or more constraints.   For example:

        •   increase the budget for sampling and analysis;
        •   increase the width of the gray region;
        •   increase the tolerable decision error rates;
        •   relax other project constraints, such as the schedule; or
        •   change the boundaries; it may be possible to  reduce sampling and analysis costs by
           changing or eliminating subgroups that will require separate decisions.

Select the most resource-effective data collection design that satisfies all of the DQOs.
Evaluate the design options based on cost and ability to  meet the DQO constraints.  Choose
the one that provides the  best balance between cost (or expected cost) and ability  to meet the
DQOs.

       The statistical concept of a power function is  extremely useful hi investigating the
performance of alternative designs. The power function is the probability of rejecting the  null
hypothesis (H0) when the null hypothesis is false (i.e., the alternative condition is  true).  If
there was no error associated with a decision, the ideal power function would be 0 if H,, were
true, and 1 if H0 were false.  Since decisions are based on imperfect data, however, it is
impossible to achieve this ideal power function.  Instead, the power function will most  likely
yield values that are small when H0 is true and large  when H0 is false.  A performance  curve
is based on the graph of the power function.1 The performance curve can be overlaid into
   'In this guidance, the performance curve is based on either the power curve or the complement of the power curve. This
ensures that the performance curve always rises from left to right

EPAQA/G-4                                    39         '                    September 1994

-------
the Decision Performance Goal Diagram to assess how well a test performs or to compare
competing tests.  A design that produces a very steep performance curve is preferred over one
that is relatively flat. An example of a performance curve is shown in Figure 7-1.
                                                            Tolerable
                                                             False
                                                            Negative
                                                 Performance Curve
                          Tolerable
                            False
                           Positive
                           Decision
                          Error Rates
           Gray Region
            (Relatively Large
             Decision Error
              Rates are
             ContWered
                                                                               0.95
                      50   I  70  I  90   I 110    130  I 150    170 I  190  I
                          60    80    100    120   140   160   180   200

                                        *•— Action Level

                    True Value of the Parameter (Mean Concentration, ppm)
                       Figure 7-1.  An Example of a Power Curve
               Baseline Condition: Parameter is Less Than Action Level.

Document the operational details and theoretical assumptions of the selected design in
the sampling and analysis plan. Document the selected design's key features that must be
implemented properly to allow for efficient and valid statistical interpretation of the data.  It
is particularly important to document the statistical assumptions that could be violated through
errors in or practical constraints on field sample collection procedures or analytical methods.

       After all the activities have been completed it may be helpful to enlist the advice and
review of a statistician with expertise in data collection designs.  This will be particularly
useful if the initial data collection designs have been developed by an inexperienced
statistician or an environmental scientist with limited statistical training.  The experienced
statistician may be able to offer innovative alternative data collection designs that may be
more cost-effective or simpler to implement.
EPA QA/G-4
40
September 1994

-------
                                BIBLIOGRAPHY
Bates, D.J., R.O. Gilbert, N.L. Hassig, R.F. O'Brien, B.A. Pulsipher.  November 1993.
       Decision Performance Criteria: The Driver Behind the Data Quality Objectives
       Process, A Statistical Introduction (Draft).  Battelle Pacific Northwest Laboratory,
       Richland, Washington.

Cochran, W.  1977.  Sampling Techniques. New York: John Wiley.

Desu, M.M., and D. Raghavarao.  1990.  Sample Size Methodology. San Diego, CA:
       Academic Press.

Gilbert, Richard O.  1987.  Statistical Methods for Environmental Pollution Monitoring.  New
       York:  Von Nostrand Reinhold.

Guenther, William C.  1977.  Sampling Inspection in Statistical Quality Control. Griffin's
       Statistical Monographs and Courses, No. 37, London:  Charles Griffin.

Guenther, William C.  1981.  "Sample Size Formulas for Normal Theory T Test." The
      American Statistician. Vol. 35, No. 4.

U.S. Environmental Protection Agency. 1994. EPA Quality System Requirements for
      Environmental Programs. EPA QA/R-1.

U.S. Environmental Protection Agency. 1994. EPA Requirements for Quality Assurance
      Project Plans for Environmental Data Operations.  EPA QA/R-5.

U.S. Environmental Protection Agency. 1994. EPA Requirements for Quality Management
      Plans.  EPA QA/R-2.

U.S. Environmental Protection Agency. 1994. Guidance for Data Quality Assessments.  EPA
      QA/G-9.

U.S. Environmental Protection Agency. 1993. Guidance for Planning in Support of
      Environmental Decision Making Using the Data Quality Objectives Process (Interim
      Final).  Quality Assurance Management Staff.
EPAQA/G-4                                41                            September 1994

-------
U.S. Environmental Protection Agency.  1992. Statistical Methods for Evaluating the
      Attainment of Cleanup Standards:  Volume III: Reference-Based Standards for Soils
      and Solid Media. EPA 230-R-94-004, Office of Policy, Planning and Evaluation.

U.S. Environmental Protection Agency.  1992. Methods for Evaluating the Attainment of
      Cleanup Standards: Volume 2: Ground Water.  EPA 230-R-92-014, Office of Policy,
      Planning and Evaluation.

U.S. Environmental Protection Agency.  1989. Methods for Evaluating Attainment of
      Cleanup Standards:  Volume 1: Soils and Solid Media. EPA 230/02-89-042, Office
      of Policy, Planning and Evaluation.

U.S. Environmental Protection Agency.  1986. Development of Data Quality Objectives,
      Description of Stages I and II. Quality Assurance Management Staff.

U.S. Environmental Protection Agency.  April 1984. "Order 5360.1, Policy and Program
      Requirements to Implement the Mandatory Quality Assurance Program." Office of the
      Administrator.
EPAQA/G-4                                  42        '                    September 1994

-------
                                 APPENDIX A

                       BEYOND THE DQO PROCESS:
              THE QUALITY ASSURANCE PROJECT PLAN
                    AND DATA QUALITY ASSESSMENT
Overview
       This appendix explains some important QA management steps that occur after the
DQO Process has been completed. The DQO Process is part of the planning phase of the
data collection operation, as illustrated in Figure A-l. At the completion of the DQO Process,
the planning team will have documented the project objectives and key performance
requirements for the data operations in the DQOs, and will have identified a data collection
design that is expected to achieve the DQOs. The data collection design and DQOs will then
be used to develop the  Quality Assurance Project Plan (QAPP), which provides the detailed
project-specific objectives, specifications, and procedures needed to conduct a successful data
collection activity.  During the implementation phase of the data collection life cycle, the
QAPP is executed and the data are collected. During the assessment phase, a Data Quality
Assessment (DQA)  is performed on the data to  determine if the DQOs have been satisfied.
The relationship between the DQO Process and these subsequent activities are explained hi
more detail below.

Quality Assurance Project Plan Development

       The QAPP is a formal EPA project document that specifies the operational procedures
and quality assurance/quality control (QA/QC) requirements for obtaining environmental data
of sufficient quantity and quality to satisfy the project objectives.  The QAPP is an important
part of the EPA Quality System, and is required for all data collection activities that generate
data for use by EPA.1 The QAPP contains information on project management, measurement
and data acquisition, assessment and oversight, and data validation and useability.

       The DQO Process may be viewed as a preliminary  step in the QAPP development
process, as shown in the right half of Figure A-l.  DQOs are a formal element of the QAPP,
yet information contained in the DQOs relates indirectly to many other elements of the
QAPP.  In essence,  the DQOs provide statements about the expectations and requirements of
the data user (such as a decision maker).  In the QAPP, these requirements are translated into
measurement performance specifications and QA/QC procedures for the data suppliers, to
   'U.S. Environmental Protection Agency. EPA Requirements for Quality Assurance Project Plans for
Environmental Data Operations. EPA QA/R-5, 1994.

EPAQA/G-4                                43        '                   September  1994

-------
provide them with the information they need to satisfy the data user's needs.  Thus, the
QAPP integrates the DQOs, the data collection design, and QA/QC procedures into a coherent
plan to be used for collecting defensible data that are of known quality and that is adequate
for the data's intended use.

PLANNING
Data Quality Objectives Process
Quality Assurance Project Plan Development
I
IMPLEMENTATION
Reid Data Collection and Associated
Quality Assurance / Quality Control Activities
I
ASSESSMENT
Data Validation
Data Quality Assessment


/-^
\
QA PLANNING FOR
DATA COLLECTION
IData Qualltv ObtaethrM Proem** \

1 OUTPUTS 1
/Data / / Data /
/ Quality / / Collection /
/ Objectives/ > / Design /
1 INPUTS 1
Quality Assurance Project Plan
Development
|
Quality
Assurance
1 Project Plan

                    Figure A-l. QA Planning and the Data Life Cycle.

       The QAPP is structured into three sections:  the Introduction, Requirements, and
Elements. The Elements are the individual requirements of the QAPP that are listed
separately. The Elements are grouped into four categories: Project Management,
Measurement/Data Acquisition, Assessment/Oversight, and Data Validation and Useability.
The outputs of the DQO Process will provide information or inputs to elements in the Project
Management section.
EPA QA/G-4
44
September 1994

-------
 Data Quality Assessment

        After the environmental data have been collected and validated in accordance with the
 QAPP, the data must be evaluated to determine whether the DQOs have been satisfied. EPA
 has developed guidance on Data Quality Assessment (DQA) to address this need (see Figure
 A-2).2  DQA involves the application of statistical tools to determine:

        •   whether the data meet the assumptions under which the DQOs and the data
           collection design were developed; and

        •   whether the total error in the data is small enough to allow the decision maker to
           use the data tc support the decision within the tolerable decision error rates
           expressed by the decision maker.

        It is important to verify the assumptions that underlie the DQOs* and the data
 collection design so that statistical calculations performed on the data relate to the decision
 maker's problem in a scientifically valid and meaningful way.  If the data do not support the
 underlying assumptions, then corrective actions must be taken to ensure that the decision
 maker's needs are met. Corrective action may be as simple as selecting a different statistical
 approach that relies on assumptions that are in better agreement with the data, or it may be as
 complicated as revising the data collection design and collecting new data that satisfy the
 decision maker's needs.

       If the data support the conclusion that the assumptions  are reasonable, then the next
 step of a DQA can be taken, which is to evaluate how well the data support the actual
 decision.  This is determined by evaluating whether the data conclusively demonstrate that the
 population parameter of interest is above (or below) the action level. In essence, this is
 where the decision maker applies a more specific or "operational" version of the decision rule
 that was developed hi Step 5 of the DQO Process (hi statistical terms, this is performing the
 hypothesis test).  Whether the data are "conclusive" or not will depend on the estimated value
 and variability of the statistical parameter hi relation to the gray region and the limits on
 decision errors that were specified hi Step 6 of the DQO Process.  If the decision cannot be
 made in accordance with the decision maker's DQOs, then the decision maker must decide
 whether to take corrective actions  (such as collect more or better data), relax the DQOs, or
 make a decision anyway, without the benefit of adequate data.
   2 U. S. Environmental Protection Agency. Guidance for Data Quality Assessments. EPA QA/G-9, 1994.

EPAQA/G-4                                   45         '                    September 1994

-------
       Thus, DQA is an essential element of the data operation because it helps to bring
closure to the issues, raised at the beginning of the DQO Process.  By verifying the
assumptions required to draw scientifically valid and meaningful conclusions from the data,
and by implementing the decision rule, DQA helps the decision maker determine whether the
DQOs have been satisfied.
           PLANNING
       Data Quality Objectives Process
   Quality Assurance Project Plan Development
                I
      IMPLEMENTATION
     Field Data Collection and Associated
   Quality Assurance / Quality Control Activities
                I
         ASSESSMENT
            Data Validation
         Data Quality Assessment
c












HJALJTY ASSURANCE ASSESSMENT
/ / /GC/feHnrrnanca 1
1 ^^ / pssss:/
I INPUTS 1
DATA VALIDATION/VERIFICATION
• uorifv maannromant narfrvmanra
• verify measurement procedures and
reporting
1 OUTPUT
I VALIDATED/VERIFIED DATA /
1 INPUT
DATA QUALITY ASSESSMENT
• verify DQOs
• verify assumptions
• make statistical decision
1 OUTPUT
I CONCLUSIONS DRAWN FROM DATA /


F











                     Figure A-2.  Quality Assurance Assessment
EPA QA/G-4
46
September 1994

-------
                                  APPENDIX B

           DQO CASE STUDY:  CADMIUM-CONTAMINATED
                                FLY ASH WASTE

Introduction

       This appendix presents a functional, but realistic example of the DQO outputs for a
decision that could be made within the Resource Conservation and Recovery Act (RCRA)
hazardous waste management program.  The example is intended to illustrate  the types of
outputs that are common to the DQO Process.  It is not intended, however, to represent the
policy of the RCRA program for actual situations that may be similar to the example.  Please
consult with a knowledgeable representative within the RCRA program office about the
current policy  for making waste classification decisions for fly ash or other types of
hazardous waste.                                                >

       The case study has been chosen because it is simple and straightforward, and because
the outputs are uncomplicated.  Although some of the outputs from this example may seem
intuitive, this is not often the case  in practice.  For many studies, the DQO Process is
complicated and thought-provoking. Even so, some steps will require more effort than others.
Keep in mind  that all of the steps in the  DQO Process are  necessary to develop a data
collection design. Once the first six steps have been completed  and thoroughly thought-out,
then development of the most resource-effective data collection design can proceed.

Background

       A waste incineration facility located in the Midwest routinely removes fly ash from its
flue gas scrubber system and disposes of it in a local sanitary landfill. Previously it was
determined that the ash was not hazardous according to RCRA program regulations. The
incinerator, however, recently began treating a new waste stream. The representatives of the
incineration company are concerned that the waste fly ash could now contain  hazardous levels
of cadmium from the new waste sources. They have decided to test the ash to determine
whether it should be sent to a hazardous  waste landfill or continue to be sent to the municipal
landfill. They  have decided to employ the DQO Process to help guide their decision making.

       Cadmium is primarily used as corrosion protection on metal parts of cars and electrical
appliances. It  is also used hi some batteries.  Cadmium and cadmium salts have toxic effects
for humans through both ingestion and inhalation exposures.  Digestion exposure usually
causes mild to  severe irritation of the gastrointestinal tract, which can be caused by
concentrations  as low as 0.1 mg/kg/day.  Chronic (long-term) inhalation exposure can cause
increased incidence of emphysema and chronic bronchitis, as well as kidney damage.
EPAQA/G-4                                 47        '                    September 1994

-------
       Under the current Code of Federal Regulations, 40 CFR, Part 261, a solid waste can
be considered "hazardous" if it meets, specific criteria of ignitability, corrosivity, reactivity,
and toxicity.  One method that is used to determine if a solid substance, such as fly ash,
meets the criteria for toxicity under the RCRA program regulations is to test a "representative
sample" of the waste and perform a Toxicity Characteristic Leaching Procedure (TCLP)
described in 40 CFR, Pt. 261, App. n.  During this process, the solid fly ash will be
"extracted" using an acid solution.  The extraction liquid (the TCLP leachate) will then be
subjected to tests for specific metals and compounds.  For this  example, the only concern is
with the concentration of cadmium  in the leachate. The primary benefit of the DQO Process
will be to establish the data collection design needed to determine if the waste is hazardous
under RCRA regulations within tolerable decision error rates.

       As a precursor to the DQO Process, the incineration company has conducted a pilot
study of the fly ash to determine the variability hi the  concentration of cadmium between
loads of ash leaving the facility.  They have determined that  each load is fairly homogeneous.
There is a high variability between  loads, however, due to the nature of, the  waste stream.
Most of the fly ash produced is not hazardous and may be disposed of in a sanitary landfill.
Thus, the company has decided that testing each individual waste load before it leaves the
facility would be the most economical.  Then they could send loads of ash that exceeded the
regulated  standards to the higher cost RCRA landfills and continue to send the others to the
sanitary landfill.

POO Development

       The following is a representative example of the output from each step of the DQO
Process for the fly ash toxicity problem.

State the Problem — a description of the problem(s)  and specifications of available
resources  and relevant deadlines for the study.

(1)    Identify the members of the planning team — The members of the planning team will
       include the incineration plant manager, a plant engineer, a statistician, a quality
       assurance officer, an EPA representative who works within the RCRA program, and a
       chemist with sampling  experience.

(2)    Identify the primary decision maker — There will not be a primary decision maker;
       decisions will be made by consensus.

(3)    Develop a concise description of the problem — The  problem is to determine which
       loads should be sent to a RCRA landfill versus a sanitary landfill.

(4)    Specify available resources and relevant deadlines for the study — While the project
       will not by constrained by cost, the waste generator (the incineration company)  wishes
       to  hold sampling costs  below $2,500.  They have also requested that the waste testing
       be completed within '1  week for each container load.

EPAQA/G-4                                   48                             September 1
-------
Identify the Decision — a statement of the decision that will use environmental data and the
actions that could result from this decision.

(1)    Identify the principal study question — Is the fly ash waste' considered hazardous
       under RCRA regulations?

(2)    Define alternative actions that could result from resolution of the principal study
       question —

           (a)    The waste fly ash could be disposed  of in a RCRA landfill.

           (b)    The waste fly ash could be disposed  of in a sanitary landfill.

(3)    Combine the principal study question and the alternative actions into a decision
       statement — Decide whether or not the fly ash waste is hazardous under RCRA and
       requires special disposal procedures.                          '

(4)    Organize multiple decisions — Only one decision is being evaluated.

Identify the Inputs to the Decision — a list of the environmental variables or characteristics
that will be measured and other information needed to resolve the decision statement.

(1)    Identify the information that will be required to resolve the decision statement — To
       resolve the decision statement, the planning team needs to obtain measurements of the
       cadmium concentration in the leachate resulting from TCLP extraction.

(2)    Determine the sources for each item of information  identified — The fly ash should be
       tested to determine if it meets RCRA regulated standards for toxicity using the test
       methods listed in  40 CFR, Pt. 261, App. II. Existing pilot study data provide
       information about variability, but do not provide enough information to resolve the
       decision statement

(3)    Identify the information that is needed to establish the action level — The  action level
       will be based on the RCRA regulations for cadmium in TCLP leachate.

(4)    Confirm that appropriate measurement methods exist to provide the necessary data —
       Cadmium can be measured  in the leachate according to the method specified in 40
       CFR, Pt. 261, App. n.  The detection limit is below the standard.

Define the Boundaries of the Study — a detailed description of the spatial and temporal
boundaries of the problem, characteristics that define the population of interest, and any
practical considerations for the study.
EPAQA/G-4                                  49        '                    September 1994

-------
(1)    Specify the characteristics that define the population of interest — Fly ash waste from
       the hazardous waste incinerator will be analyzed. The fly ash should not be mixed
       with any other constituents except water that is used for dust control. Each load of
       ash should fill at least 70% of the waste trailer.  In cases where the trailer is filled less
       than 70%, the trailer must wait on-site until more ash is produced and fills the trailer
       to the appropriate capacity.

(2)    Define the spatial boundary of the decision statement —

       (a) Define the geographic area to which the decision statement applies. Decisions
          will apply to each container load of fly ash waste.

       (b) When appropriate, divide the  population into strata that have relatively
          homogeneous characteristics.   Stratification is not necessary since the waste ash is
          relatively homogeneous within each container.
                                                                  i
(3)    Define the temporal boundary of the decision statement —

       (a) Determine the timeframe to which the decision statement applies.  It will be
          assumed that the sampling data represent both the current and future concentration
          of cadmium within the ash.

       (b) Determine when to collect data.  Contained hi the trucks, the waste does not pose
          a threat to humans or the environment.  Additionally, since the fly ash is not
          subject to change, disintegration, or alteration, the decision about the waste
          characteristics does not warrant any temporal constraints. To expedite decision
          making, however, the planning team has placed deadlines on sampling and
          reporting. The fly ash waste will be tested within 48 hours of being loaded onto
          waste hauling trailers. The analytical results from each sampling round should be
          completed and reported within 5 working days of sampling.  Until analysis is
          complete, the trailer cannot be used.

(4)    Define the scale of decision making — The  scale of  decision making will be each
       container of waste ash.

(5)    Identify practical constraints on data collection — The most important practical
       consideration that could interfere  with the study is the ability to take samples from the
       fly ash that is stored in waste hauling trailers. Although the trailers have open access.
       special procedures and methods will have to be implemented for the samples to be
       representative of the entire depth  of the ash.  It has been suggested that core samples
       may be one practical  solution to this problem.  To get additional samples from each
       truck and to minimize the cost, compositing of core samples has been suggested.
EPAQA/G-4                                   50        '                     September 1994

-------
Develop a Decision Rule — to define the parameter of interest, specify the action level and
integrate previous DQO outputs into a single statement that describes a logical basis for
choosing among alternative actions.

(1)    Specify the statistical parameter that characterizes the population of interest — The
       planning team is interested in the true mean concentration of cadmium in the TCLP
       leachate for each container.

(2)    Specify the action level for the study — The  action level for the decision will be the
       RCRA regulatory standard for cadmium of 1.0  mg/L in the TCLP leachate.

(3)    Develop a decision rule (an "if...then..." statement) — If the mean concentration of
       cadmium from the fly ash leachate in each container load is greater than 1.0 mg/L
       (using the TCLP method as defined in 40 CFR 261), then the waste will be considered
       hazardous and will  be disposed of at a RCRA landfill.  If the mean concentration of
       cadmium from the fly ash waste leachate is less than 1.0 mg/L ("using the TCLP
       method as defined in 40 CFR 261), then the waste will  be considered non-hazardous
       and will be disposed of in a sanitary landfill.

Specify Tolerable Limits on Decision Errors — the decision  maker's tolerable decision
error rates based on a consideration of the consequences of making a decision error.

(1)    Determine the possible range of the parameter  of interest — From analysis of records
       of similar studies of cadmium in environmental matrices, the range of the cadmium
       concentrations is expected to be from 0-2 mg/L. Therefore the mean concentration is
       expected to be between 0-2 mg/L for this investigation.

(2)    Identify the decision errors and choose the null hypothesis —

       (a) Define both  types of decision errors and establish the true state of nature for each
          decision error.  The planning team has determined that the two decision errors are
          (i) deciding that the waste is hazardous when it truly is not, and (ii) deciding that
          the waste is  not hazardous when it truly is.

              The true state of nature for decision error (i) is that the waste is not hazardous.
                                                                                   /
              The true state of nature for decision error (ii) is that the waste is hazardous.

       (b) Specify and evaluate the potential consequences of each decision error.

              The consequences  of deciding that the waste is hazardous when it truly is not
              will be that the incinerator company will have to pay more for the disposal  of
              the fly ash at a RCRA facility than at a sanitary  landfill.
EPAQA/G-4                                   51         •                     September 1994

-------
              The consequences of deciding that the waste is not hazardous when it truly is
              will be that the incinerator company will dispose of the waste in a sanitary
              landfill which could possibly endanger human health and the environment.  In
              this situation, they may also be liable for future damages and environmental
              cleanup costs. Additionally, the reputation of the incinerator company may be
              compromised, jeopardizing its future profitability.

       (c) Establish which decision error has more severe consequences near the action
          level.  The planning team has concluded that decision error (ii) has the more
          severe consequences  near the action level since the risk of jeopardizing human
          health outweighs the consequences of having to pay more for disposal.

       (d) Define the null hypothesis (baseline condition) and the alternative hypothesis and
          assign the terms "false positive" and "false negative" to the appropriate decision
          error.
                                                                   j
              The baseline condition or null hypothesis (HJ is "the waste is hazardous."

              The alternative hypothesis (H,)  is "the waste is not hazardous."

          The false positive decision error occurs when the null hypothesis is rejected when
          it is true. For this example,  the false positive decision error occurs when die
          decision maker decides die waste is not hazardous when it truly is hazardous. The
          false negative decision error  occurs when the null hypothesis is not rejected when
          it is false. For this example, the false negative decision error occurs when the
          decision maker decides that the  waste is hazardous when it truly is not hazardous.

(3)    Specify a range of possible values of the parameter of interest where the consequences
       of decision errors are relatively  minor (gray region)  — The gray region is the area
       adjacent to the action level where the planning team  feels that the consequences of a
       false negative decision error are  minimal.  To decide how to set the width of the gray
       region, the planning team must decide  where the consequences of a false negative
       decision error are minimal. Below the action level, even if the concentration of
       cadmium were  very close to the  action level, the monetary costs of disposing of the
       waste at a RCRA facility are the same as if the waste had a much lower concentration
       of cadmium. Clearly any false negative decision error (to the left of the action level)
       will cause the incinerator company and their customers to bear the cost of unnecessary
       expense (i.e., sending nonhazardous waste to a RCRA facility). The planning team,
       however, also realizes that they must define a reasonable gray region that balances the
       cost of sampling with risk to human health and the environment and the ability of
       measurement instruments to detect differences.  Therefore the planning team has
       specified a width of 0.25 mg/L for this gray region based on their preferences to detect
       decision errors at a concentration of 0.75 mg/L (see Figure B-l).
EPAQA/G-4                                   52         '                    September 1994

-------
(4)    Assign probability values to points above and below the action level that reflect the
       tolerable probability for the occurrence of decision errors — For this example, RCRA
       regulations allow a 5% decision error rate at the action level.  The planning team has
       set the decision error rate to 5% from  1 mg/L to 1.5 mg/L and 1% from 1.5 mg/L to 2
       mg/L as the consequences of health effects from the waste disposed of in the
       municipal landfill increase. On the other side of the action level, the planning team
       has set the tolerable probability of making a false negative error at 20% when the true
       parameter is from 0.25 to 0.75 mg/L and 10% when it is below 0.25 mg/L, based on
       both experience and an economic  analysis that shows that these decision -error rates are
       reasonable to balance the cost of sampling versus the consequence of sending clean
       ash to the RCRA facility (see Figure B-l).

Optimize the Design — select the most resource-effective data collection and analysis design
for generating data that are expected to satisfy the DQOs.  Optimizing the design is the one
step of the DQO Process that will  most likely be completed by a statistician or someone who
has data collection design expertise. Using the case study as an example, the following
section has been included to provide the reader with a background on the overall process that
the statistician might follow to optimize the final data collection design.
                                                           Tolerable
                                                             False
                                                            Positive
                                                            Decision
                                                           Error Rates
                                                  Performance
                                                      Curve
                                                     Gray Region
                                  1.0

                                  0.9

                                  0.8

                                  0.7

                                  0.6

                                  0.5

                                  0.4

                                  0.3

                                  0.2

                                  0.1
                       •25    .50    .75    1.0   1.25     1.5   1.75    2.0

                                            *	Action Level
                  True Value of the Parameter (Mean Concentration, mg/L)
Figure B-l.  Decision Performance Goal Diagram for Cadmium Compliance Testing
                    Baseline Condition:  Mean Exceeds Action Level.
EPA QA/G-4
53
September 1994

-------
Overview

       Developing a data collection design requires an understanding of the sampled medium
and the information that was generated in previous DQO steps. The statistician's job is to
review the background information, determine the appropriate statistical application to
adequately solve the problem, and develop one or more appropriate data collection designs.
Once this is complete, the statistician will compare the cost and performance of the different
data collection designs.  This process can be broken down into five distinct steps:

       (1)    Review the DQO outputs and existing environmental data.

       (2)    Develop general  data collection design alternatives.

       (3)    For each data collection design alternative,  select the optimal sample size that
              satisfies the DQOs.
                                                                  j
       (4)    Select the  most resource-effective data collection design that satisfies all of the
              DQOs.

       (5)    Document the operational details and theoretical assumptions of the selected
              design hi the sampling and analysis plan.

Activities

(1)    Review the DQO outputs and existing environmental data — Because the statistician
       has participated hi the DQO Process for this problem, there is no need to review the
       DQO outputs further.  The only existing data relevant to this problem are the  pilot
       study data. Based on the pilot study, the incineration  company has determined that
       each load of ash is fairly homogeneous, and has estimated the standard deviation hi
       the concentration of cadmium within loads of ash to be 0.6 mg/L.

(2)    Develop general data collection design alternatives — Generally, the design
       alternatives are based on a combination of design objectives developed hi previous
       DQO Process steps and  knowledge of statistical parameters about the medium or
       contaminant.  Below are four examples of possible designs that could apply to the case
       study:

       (a) Simple Random Sampling — The simplest type of probability sample is the simple
          random sample. With this type of sampling, every possible point hi the sampling
          medium has an equal chance of being selected. Simple random samples are used
          primarily when the variability of the medium is relatively small and the cost of
          analysis is relatively inexpensive. Simple random sample locations are generally
          developed through the use of a random number table or through computer
          generation of pse.udo-random numbers.

EPAQA/G-4                                 54         '                   September 1994

-------
           In the case of the cadmium-contaminated ash, a fixed number of random grab
           samples would be selected and analyzed. Standard lab splits and QC samples
           would be taken according to standard procedures for the RCRA program.  Each
           sample would be chosen randomly in three dimensions.' A Student's t-test is
           suggested as a possible method for testing the statistical hypothesis.

       (b) Composite Simple Random Sampling (composite sampling) — This type of
           sampling consists of taking multiple samples, physically combining (compositing)
           them, and drawing one or more subsamples for analysis. Composite samples are
           taken primarily when an average concentration is sought and there is no need to
           detect peak concentrations.  By compositing  the samples, researchers are able to
           sample a larger number of locations than if compositing was  not used, while
           reducing the cost of analysis by combining several samples.

           In the case of the cadmium-contaminated ash, a fixed number of random grab
           samples would be taken and composited. The number of grab samples contained
           in a composite sample (g) is also fixed.  To determine sampling locations  within
           the composite, a container would be divided  into "g" equal-volume strata and
           samples would be chosen randomly within each strata.  The use of strata ensure
           full coverage of each  container. Standard lab splits and QC samples would be
           taken according to standard procedures for the RCRA program.  A Student's t-test
           is suggested  as the possible method for testing the statistical hypothesis.

       (c) Sequential Sampling — Sequential sampling  involves making several rounds of
           sampling and analysis.  A statistical test is performed after each analysis to arrive
           at one of three possible decisions: reject the null hypothesis, accept the null
           hypothesis,1 or collect more samples. This strategy is applicable when sampling
           and/or analysis costs are high, when information concerning sampling and/or
           measurement variability is lacking, when the waste and site characteristics of
           interest are stable over the timeframe of the sampling effort, and when the
           objective of the sampling is to test a single hypothesis.  By taking samples in
           sequence, the researcher can hold down the cost of sampling  and analysis.

           In the case of the cadmium-contaminated ash, a sequential probability sample
           could be performed.  The samples in each sampling round would be chosen
           randomly in three dimensions. If the decision to stop sampling has not been made
           before the number of  samples required for the simple random sample are taken,
           sampling would stop at this point and the simple random sample test would be
           performed. Standard  laboratory splits and QC samples would be taken according
           to standard procedures for the RCRA program.  An approximate ratio test  is
'Decide not to reject the null based on tolerable decision error limits.

EPAQA/G-4                                  55         •                    September 1994

-------
          suggested after each round of sampling is complete to decide whether or not to
          conclude that the waste is hazardous or to continue sampling.

       (d) Stratified Random Sampling — Stratified sampling involves dividing the study
          area into two or more non-overlapping subsets (strata) which cover the entire
          volume to be sampled. These strata should be defined so that physical samples
          within a stratum are more similar to each other than to samples from other strata.
          Sampling depth, concentration level, previous cleanup attempts, and confounding
          contaminants can be used as the basis for creating strata.  Once the strata have
          been defined, each stratum is then sampled separately using one of the above
          designs. Stratification is often used to ensure that important areas of a site are
          represented in the sample. In addition, a stratified random sample may provide
          more precise estimates of contaminant levels than those obtained from a simple
          random sample. Even with imperfect information, a stratified  sample can be more
          resource-effective.
                                                                  t
          Since the incineration company has  already determined that each load of ash is
          fairly homogeneous, stratification does not have any advantages over a simple
          random sample. In addition, since the company has decided to test each waste
          load individually before it leaves the facility, stratifying each waste load would be
          difficult and unnecessary. Therefore, this data collection design will not  be
          considered further.

(3)     For each data collection design alternative,  select the optimal sample size that
       satisfies the DQOs — The formula for determining the sample size (number of
       samples to be collected) is chosen based on the hypothesis test and data collection
       design.  Standard formulas can be found in several references, including:

          •  Cochran, W.  1977. Sampling Techniques. New York:  John Wiley.

          •  Desu, M.M., and D. Raghavarao.  1990.  Sample Size Methodology. San Diego,
                CA: Academic Press.

          •  Gilbert, Richard O.  1987.  Statistical Methods for Environmental Pollution
                Monitoring. New  York:  Van Nostrand Reinhold.

          •  U.S. Environmental Protection Agency.  1989. Methods for Evaluating the
                Attainment of Cleanup Standards:  Volume 1: Soils and Solid Media.
                EPA 230/02-89-042, Office of Policy, Planning and Evaluation.

          •  U.S. Environmental Protection Agency.  1992. Methods for Evaluating the
                Attainment of Cleanup Standards:  Volume 2: Ground Water.
                EPA 230-R-92-014, Office of Policy, Planning and Evaluation.
EPAQA/G-4                                   56        '                    September 1994

-------
          •   U.S. Environmental Protection Agency.  1994. Statistical Methods for
                Evaluating the Attainment of Clean-up Standards: Volume 3: Reference-
                Based Standards for Soils and Solid Media. EPA 230-R-94-004.  Office of
                Policy, Planning and Evalutaion.

       These formulas can also be found in many basic statistics textbooks.  Different
       formulas are necessary for each data collection design, for each parameter, and for
       each statistical test. These formulas are generally a function of a; p; the detection
       difference,  A (delta); and the standard deviation, a.  The detection difference, A, is
       defined to be the difference between the action level (AL) and the other bound of the
       gray region (U); i.e., A = AL - U.  In this case the standard deviation was derived
       from pilot data under approximately the same conditions as expected for the real
       facility.

       For example, a formula for computing the sample size necessary to meet the DQO
       constraints  for comparing a mean against a regulatory threshold,' when a simple
       random sample is selected, is:

                               n =
                                        A2
       where:

          d2 =  estimated variance in measurements (from pilot study)
          n  =  number of samples required,
          Zp =  the p* percentile of the standard normal distribution (from standard
                statistical tables), and
          A  =  U-AL

       Simple Random Sample — Using the formula above, it was determined that 37
       samples are necessary to achieve the specified limits on decision errors.  This
       sampling plan satisfies all the DQOs including budget, schedule, and practical
       constraints.

       Composite Sampling — To determine sample sizes for a composite sample, it is
       necessary to compute the number of composites samples, n; the number of samples, g,
       within each composite; and the number of subsamples, m, to be measured for each
       composite.  Usually m=l; however, since this design is to be used repeatedly, it is
       suggested that two subsamples from each composite sample be measured to estimate
       composite variability, which can then be used to re-optimize the number of samples m
       and g.  .

       For a composite sample, with random sample locations, it has been determined  that
       eight composite samples of eight samples each are sufficient to meet the limits on
       decision errors that have been specified. This design is more than sufficient to

EPAQA/G-4                                  57        •                    September 1994

-------
       achieve the specified limits on decision errors and satisfies all the DQOs including
       budget, schedule, and practical constraints.

       Sequential Sampling — For the purposes of comparing costs, the average number of
       samples in a sequential sampling design can be estimated, but these estimates are only
       averages. The average sample size for concluding that the waste is hazardous is  16
       and the average sample size for concluding the waste is not hazardous is 22. The
       average sizes are different because the burden of proof is placed on disproving the null
       hypothesis, thus, more samples on average are required to prove that the alternative
       hypothesis (the waste is not hazardous)  is true. However, these sample sizes are  only
       averages. In some cases, fewer samples are necessary; in others, more may be
       necessary. This  sampling plan satisfies all the DQOs including budget, schedule, and
       practical constraints.

(4)    Select the most resource-effective data collection design that satisfies the DQOs —
       Compare the overall efficiency of each model and choose the one that will solve  the
       problem most effectively.

       Cost Estimates for Each Design

       First, the costs for the three designs alternatives will be evaluated:

       Simple Random  Sampling — A  simple  random sampling scheme can be implemented
       for each load of  fly ash by first generating three-dimensional random sampling points.
       This can most easily be done by using a computer.  Samples can then be taken using a
       special grab sampler which will be forced into the ash, opened to take the sample,
       then closed and removed.  The difficulty with this type of sampling scheme is
       measuring sampling locations in three dimensions, and it may be difficult to gain
       access to the correct sampling locations.

       This design meets all of the required limits on decision errors. The cost of this design
       is calculated based on the assumed cost of selecting a sample ($10), and the cost  of
       analyzing a sample ($150). Since 37 samples need to be taken and analyzed, the cost
       of this  design is:

          CostsRs    =  37 x $10 + 37 x $150
                    =  $370 +  $5550 = $5920

       Composite Sampling — Composite sampling will be performed similarly to simple
       random sampling except that after eight random samples are collected (one from each
       stratum), they will be combined and homogenized.  Two sample aliquots for analysis
       will then be drawn from the homogenized mixture.  This process will be repeated
       eight times.
EPAQA/G-4                                  58        '                    September 1994

-------
       This design meets all of the required limits on decision errors.  The cost of this design
       is based on the cost of selecting ($10) and analyzing ($150) a sample.  Eight samples
       will be used .to make each composite sample for a sampling cost of $80; two
       subsamples will be analyzed from this composite sample for a cost of $300.
       Therefore, each composite sample will cost $380.  The total cost of this design is:

           Costa,    = 8 x $380 = $3040.

       Sequential Sampling — Sequential sampling will be performed similarly to random
       sampling. The primary difference is that the ultimate number of samples will be
       determined by the results of one or more sampling rounds.

       This design has the potential to reduce the number of samples required in the simple
       random sampling design and still meet the decision error limits. The average  costs of
       the two decisions are used below:
                                                                 j
           The ash is hazardous:          16 x ($160) = $2,560
           The ash is non-hazardous:      22 x ($160) = $3,520

       To determine the expected cost, estimate the number of loads of ash that should be
       sent to a  RCRA facility versus the number of loads that can be sent to a municipal
       facility.  Suppose 25% of the loads are hazardous and should be sent to a RCRA
       facility. Then the expected cost (ECSS) of this design should be

           ECSS  =    0.25 x (cost of sampling when ash is hazardous) + (0.75 x cost of
                    sampling when ash is non-hazardous)

                =   0.25 x ($2,560) + 0.75 x ($3,520) = $ 3,280

       Selection of a Design

       Because the simple random sampling design requires that many samples be taken and
       analyzed, it is inefficient for the goals of this study.  Sampling will cost almost as
       much to determine whether the waste is hazardous or nonhazardous  as it would cost to
       send all the waste to a RCRA hazardous waste landfill.  Therefore, this decision is not
       resource-effective.

       The sequential data collection.design is more resource-effective than the simple
       random sampling design.  The potential savings over sending all waste to a RCRA
       hazardous waste facility is $6,750 - $3,280 =  $3,470.  The site owner has expressed
       disapproval for this sampling plan because of the time it may take before a decision
       can be made.  If the ash was not homogeneous within a container, however, this data
       collection design may be the design of choice.
EPAQA/G-4                                  59        '                    September 1994

-------
       The composite sample design is the best option. It is the most resource-effective
       design and requires the least amount of time to implement. In addition, the use of
       strata ensures full coverage of each container. It is recommended that each of the
       eight composite samples have two subsamples analyzed.  In the future, after sufficient
       data have been collected to estimate the variability within each composite sample, it
       may be possible to reduce the number of samples that will be necessary to make a
       decision about the waste contents.

(5)    Document the operational details and theoretical assumptions of the selected design in
       the sampling and analysis plan — A composite sample design should be used to
       determine whether each container of ash should be sent to a RCRA landfill or to a
       municipal landfill. Eight composite samples, consisting of eight grab samples, should
       be taken from each container and two subsamples from each composite should be
       analyzed at the laboratory.  To  form the composite samples, the containers will be
       divided into eight strata of equal size and one grab sample will be taken randomly
       within each stratum and composited. Sample locations will be generated randomly
       using computer-generated random numbers.  The model assumes that the variability
       within a composite sample is negligible. Data from the subsamples can be used to test
       this assumption and make corrections to the model.

Beyond the POO Process • Evaluation of the Design using the DO A Process

       For this study, the data were collected using the composite sampling design. Once the
samples were collected and analyzed, the data were evaluated statistically and scientifically
using the DQA Process to inspect for anomalies, confirm that the model assumptions were
correct, select a statistical test, and verify that the test assumptions such as distribution and
independence can be met.  For this study, a t-test  satisfied the DQOs, and inspection of the
data indicated that there was no reason to believe  that the data were not normally distributed
or that there was correlation between data points.  It was also verified that the within-
composite variability was negligible.

       After three weeks of sampling,  approximately 30% of the waste loads leaving the
incinerator were found to have hazardous concentrations of cadmium in the fly ash. The data
collection design was determined to be cost-effective because the combined cost of sampling
and disposal was less than sending all  of the waste to a RCRA landfill.
EPAQA/G-4                                  60                            September 1994

-------
                                APPENDIX C

  DERIVATION OF SAMPLE SIZE FORMULA FOR TESTING MEAN
      OF NORMAL DISTRIBUTION VERSUS AN ACTION LEVEL
      This appendix presents a mathematical derivation of the sample size formula used in
the DQO example of Appendix B.
      Let Xt, X2,...^ denote a random sample from a normal distribution with unknown
mean u and known standard deviation a. The decision maker wishes to test the null
hypothesis HQ: u = AL versus the alternative HA: u > AL, where AL, the action level, is some
prescribed constant; the false positive (Type I) error rate is a (i.e., probability of rejecting HQ
when u = AL is a); and for some fixed constant U > AL (where U is the other bound of the
gray region), the false negative (Type n) error rate is (J (i.e., probability of rejecting HO when
u - U is l-(3).  Let X denote the sample mean of the Xs. It will have a normal distribution
with mean u and variance cVn.  Hence the random variable Z defined by
will have a standard normal distribution (mean 0, variance 1).  Let Zp denote the p* percentile
of the standard normal distribution (available in most statistics books).  Recall that the
symmetry of the standard normal distribution implies that z,, = -z,.p.


Case 1: Standard Deviation Known

      The test of HQ versus HA is performed by calculating the test statistic
If T > z,^, the null hypothesis is rejected.

Note that
EPAQA/G-4                               61                          September 1994

-------
where

                                   e(u) = C                                         (4)
                                              Zl Ju=AL] = Pr[Z+e(AL»z, J  = Pr[Z>Zl J  - a.  (5)

Achieving the desired power 1-|3 when u = U requires that

                                Pr[reject HQ\\i=U] = 1 -J3.


Therefore,

             Pr[r40), the
approximation is good.  The particular noncentral t distribution involved in the calculation
depends on the sample size -n. Thus, determining the exact minimum n that will satisfy the

EPAQA/G-4                                   62        •                     September 1994

-------
Type I and Type n error rate conditions requires an iterative approach in which the
noncentral t probabilities are calculated for various n values until the desired properties are
achieved.  With the .aid of a computer routine for calculating such probabilities, this is not
difficult; however, a simple and direct approach for approximating' n is available.  This
approach, whose derivation is described in the paragraphs below, leads to the following
approximate but very accurate formula for n:
                               „  .
                                                                                   (8)
 In practice, since a is unknown, a prior estimate of it must be used hi (8).
       The approach is based on the assumption that, for a given constant k, the statistic
 X-kS is approximately normal with mean u-ka and variance (oVnX 1+1^/2) (Guenther, 1977
 and 1981).
                                                                  i
       The classical t-test rejects HQ when T = [(X - AL)/(S/Vn)] > D, where the critical
 value D is chosen to achieve the desired Type I error rate a.  The inequality can be
 rearranged as X-kS>AL, where k = D/Vn. Subtracting the mean (assuming HQ) and dividing
 by the standard deviation of X-kS on both sides of the inequality leads to
                       X-kS-(AL-ka)     AL-(AL-ka)   =
By the distributional assumption on X-kS, the left side of (9) is approximately standard
normal when u = AL, and the condition that the Type I error rate is a becomes

                                                                                  (10)
                                                    = a,
One can show that (11) is equivalent to

                                                                                  (12)
The condition that the Type n error rate is P (or that power is 1-|3) when u = U means that
the event of incorrectly  accepting HO given X-kS < AL should have probability fi.
Subtracting the mean (U - ka) and dividing by the standard deviation of X-kS on both sides
of this inequality yields
EPAQA/G-4                                   63        '                     September 1994

-------
                                              AL-(U-k<5)
       Again, the left side is approximately standard normal and the Type n error rate
condition becomes
                       Pr[Z<[AL-((7-fca)]/[(o/Vn)/Vl+fe2/2]j = p,

which implies
                                           (AL-U)+ka
                                       -•                       (14)
Subtracting (14) from (11) yields

                             .   _    _     (U-AL)
or
Substituting (12) into the denominator on the right side of (16) yields
                                                                                (15)
                                                                                (16)
                               (t/-AL)

Squaring both sides of (17) and solving for n yields equation (8).

References

Guenther, William C.  1977.  Sampling Inspection in Statistical Quality Control. Griffin's
       Statistical Monographs and Courses, No. 37, London:  Charles Griffin.

Guenther, William C.  1981.  "Sample Size Formulas for Normal Theory T Test." The
       American Statistician. Vol. 35, No. 4.


EPAQA/G-4                                  64        '                    September 1994

-------
                                    APPENDIX D

                             GLOSSARY OF TERMS

action level:  the numerical value that causes the decision maker to choose one of the
       alternative actions (e.g., compliance or noncompliance). It may be a regulatory
       threshold standard, such as a Maximum Contaminant Level for drinking water; a risk-
       based concentration level; a technological limitation; or a reference-based standard.
       [Note: the action level is specified during the planning phase of a data collection
       activity; it is not calculated from the sampling data.]

alternative hypothesis:  See hypothesis.

bias: the systematic or persistent distortion of a measurement process which causes errors in
       one direction (i.e., the expected sample measurement is different than the sample's
       true value).                                                 '

boundaries:  the spatial and temporal conditions and practical constraints under which
       environmental data are collected.  Boundaries specify the area or volume (spatial
       boundary) and the tune period (temporal boundary) to which the decision will apply.
       Samples are then collected within these boundaries.

data collection design: A data collection design specifies the configuration of the
       environmental monitoring effort to satisfy the DQOs. It includes the types of samples
       or monitoring information to be collected; where, when, and under what conditions
       they should be collected; what variables are to be measured; and the Quality
       Assurance and Quality Control (QA/QC) components that ensure acceptable sampling
       design error and measurement error to meet the decision error rates specified in the
       DQOs. The data collection design is the principal part of the QAPP.

Data Quality Assessment (DQA) Process:  a statistical and scientific evaluation of the data
       set to assess the validity and performance of the data collection design and statistical
       test, and to establish whether a data set is adequate for its intended use.

Data Quality Objectives (DQOs):  Qualitative and quantitative statements derived from the
       DQO  Process that clarify study objectives, define the appropriate type of data, and
       specify the tolerable levels of potential decision errors that will be used as the basis
       for establishing the quality and quantity of data needed to  support decisions.

Data Quality Objectives Process:  a  Quality Management tool based on the Scientific
       Method, developed by the U.S. Environmental Protection Agency to facilitate the
       planning of environmental data collection activities.  The DQO Process enables
       planners to focus their planning efforts by specifying the intended use of the data (the
       decision), the decision criteria (action level), and the  decision maker's tolerable
       decision error rates.  The products of the DQO Process are the DQOs.

EPAQ.VG-4                                  65                            September 1994

-------
decision error:  an error made when drawing an inference from data in the context of
       hypothesis testing, such that variability or bias in the data mislead the decision maker
       to draw a conclusion that is inconsistent with the true or actual state of the population
       under study. See also false negative decision error, false positive decision error.

defensible: the ability to withstand any reasonable challenge related to the veracity, integrity,
       or quality of the logical, technical, or scientific approach taken in a decision making
       process.

false negative decision error:  a false negative decision error occurs when the decision
       maker does not reject the null hypothesis when the null hypothesis actually is false.
       In statistical terminology, a false negative decision error is also called a Type n error.
       The measure of the size of the error is expressed as a probability, usually referred to
       as "beta ((3)"; this probability is also called the complement of power.

false positive decision error:  a false positive decision error occurs when a decision maker
       rejects the null hypothesis when the null hypothesis actually is true.  In statistical
       terminology, a false positive decision error is also called a Type I error. The measure
       of the  size of the error is expressed as a probability, usually referred to as "alpha (a),"
       the "level of significance,"  or "size of the critical region."

gray region:  a range of values of the population parameter of interest (such as mean
       contaminant concentration) where the consequences of making a decision error are
       relatively minor.  The gray region  is bounded on one  side by the action level.

hypothesis:  a tentative assumption made to  draw out and test its  logical or empirical
       consequences.  In hypothesis testing, the hypothesis is labeled "null" or "alternative",
       depending on the decision maker's concerns for making a decision error.

limits on decision errors:  the tolerable decision error probabilities established by the
       decision maker.  Potential economic, health, ecological,  political, and social
       consequences of decision errors should be considered when setting the limits.

mean:  (i) a measure of central tendency of the population (population mean),  or (ii) the
       arithmetic average of a set  of values (sample mean).

measurement error:  the difference between the true or actual state and that which is
       reported from measurements.

median:  the  middle value for an ordered set of n values; represented by the central value
       when n is odd or by  the average of the two most central values when n is even. The
       median is the 50th percentile.

medium:  a substance (e.g., air, water, soil) which serves as  a carrier of the analytes of
       interest.

EPAQA/G-4                                    66                              September 1994

-------
natural variability:  the variability that is inherent or natural to the media, objects, or people
       being studied.

null hypothesis: See hypothesis.

parameter:  a numerical descriptive measure of a population.

percentile: the specific value of a distribution that divides the distribution such- that p
       percent of the distribution is equal to or below that value.  Example for p=95:  "The
       95th percentile is X" means that 95% of the values in the population (or statistical
       sample) are less than or equal to X.

planning team:  the  group of people that will carry out the DQO Process.  Members include
       the decision maker (senior manager), representatives of other data users, senior
       program and technical staff, someone with statistical expertise, and a QA/QC advisor
       (such as a QA Manager).

population:  the total collection of objects, media, or people to be studied and from which a
       sample is to be drawn.

power function: the probability of rejecting the null hypothesis (HJ over the range of
       possible population parameter values.  The power function is used to assess the
       goodness of a hypothesis test or to compare two competing tests.

quality assurance (QA):  an integrated system of management activities involving planning,
       quality control, quality assessment, reporting,  and quality improvement to ensure that a
       product or service (e.g., environmental data) meets defined standards of quality with a
       stated level of confidence.

Quality Assurance Project Plan (QAPP):  a formal technical document containing the
       detailed QA, QC and other technical procedures for assuring the quality of
       environmental data prepared for each EPA environmental data collection activity and
       approved prior to collecting the data.

quality control (QC):  the overall system of technical activities that measures the  attributes
       and performance of a process, item, or service against defined standards to  verify that
       they meet the  stated requirements established by the customer.

Quality Management Plan (QMP): a formal document describing the management policies,
       objectives, principles, organizational authority, responsibilities, accountability, and
       implementation protocols of an  agency, organization, or laboratory for ensuring quality
       in its products and utility to its users. Tn EPA, QMPs are submitted to the Quality
       Assurance Management Staff (QAMS) for approval.

range: the numerical difference between the minimum and maximum of a set of values.

EPAQA/G-4                                  67                              September 1994

-------
'sample:  a single item or specimen from a larger whole or group, such as any single sample
       of any medium (air, water, soil, etc.).

2sample:  a set of individual samples (specimens or readings), drawn from a population,
       whose properties are studied to gain information about the whole.

sampling: the process of obtaining representative samples and/or measurements of a subset
       of a population.

sampling design error: the error due to observing only a limited number of the total
       possible values that make up the population being studied.  It should be distinguished
       from errors due to imperfect selection; bias in response; and errors of observation,
       measurement, or recording, etc.

scientific method: the principles and processes regarded as necessary for scientific
       investigation, including rules for concept or hypothesis formulation, conduct of
       experiments, and validation of hypotheses by analysis of observations.

standard deviation:  the  square root of the variance.

statistic:  a function of the sample  measurements; e.g., the sample mean or standard
       deviation.

statistical test:  any statistical method that is used to determine which of several hypotheses
       are true.

total study error:  the combination of sampling design error and measurement error.

true: being in accord with the actual state of affairs.

Type I error:  A Type I error occurs when a decision maker rejects the null hypothesis when
       it is actually true.  See false positive decision error.

Type n error:  A Type n error occurs when die decision maker fails to reject the null
       hypothesis when it is actually false.  See false negative decision error.

variable:  The attribute of the environment that is indeterminant.

variance:  a measure of (i) the variability or dispersion in a population (population variance),
       or (ii) the sum of the squared  deviations of the measurements about their mean divided
       by the degrees of freedom (sample variance).
EPAQA/G-4                                   68                             September J994

-------