PROCESS FOR SELECTING INDICATORS


     AND DATA AND FILLING INFORMATION GAPS




                  Final Report
WESTAT
An  Employee-Owned Research Corporation

-------
K:\12oaks\do26\finalrpt.wp5- ah - July 14, 1994
                      PROCESS FOR SELECTING INDICATORS

                   AND DATA AND FILLING INFORMATION GAPS


                                    Final Report
                                    Prepared by:

                                    Westat, Inc.
                                 1650 Research Blvd.
                                 Rockville, MD 20850
                                    Prepared for:

                          U.S. Environmental Protection Agency
                                  401 M Street, S.W.
                                Washington, DC 20460
                             This work was conducted under
                           EPA Contract Number 68-W1-0019
                                    July 14, 1994

-------
                             TABLE OF CONTENTS


Section                                                                    Page

INTRODUCTION  	     1

DEFINITION OF ENVIRONMENTAL INDICATOR	     1

FRAMEWORKS FOR DEVELOPING INDICATORS	     2

GENERAL COMMENTS ON THE PROCESS OF SELECTING
ENVIRONMENTAL INDICATORS	     4

CRITERIA FOR SELECTING INDICATORS FOR ASSESSING
ENVIRONMENTAL STATUS AND TRENDS  	'.	     6

CRITERIA FOR SELECTING EXISTING DATA SETS TO QUANTIFY INDICATORS . .    8

PROCESS FOR SELECTING INDICATORS AND DATA	 .    10

      Step 1.   Identify and Recommend Possible Indicators for Reporting Status
              and Trends	    10
      Step 2.   Inventory and Describe Existing Data Sets that may be Suitable for
              Quantifying Indicators	    12
      Step 3.   Identify and Recommend Information/Data for Selected Indicators  	   12
      Step 4.   Fill Information Gaps	    13


                                 List of Tables

Table

  1      Criteria for Selecting Indicators	     7

  2      Criteria for Selecting Existing Data Sets to Quantify Indicators	     9


                                 List of Figures

Figure

  1      OECD Pressure-State-Response Framework	     3

  2      Continuum of Environmental Indicators  	     5

  3      Process for Selecting Indicators and Data and Filling Gaps	    11
                                       11

-------
                          TABLE OF CONTENTS (continued)






                                  List of Appendices






Appendix                                                                      Page



 A      Draft Short Form for Screening Candidate Data Sets  	  A-l
                                        in

-------
WeiUt, July 14,1994
                         PROCESS FOR SELECTING INDICATORS
                                          AND DATA
                            AND FILLING INFORMATION GAPS
INTRODUCTION

The purpose of this document is to present a process for selecting indicators and data sets that can be used
to measure the current status of the environment and to show patterns or trends in that status. This
proposed process is directed primarily to technical managers within EPA who have responsibility for the
specification and quantification of indicators.


DEFINITION OF ENVIRONMENTAL INDICATOR

Many definitions of environmental and environmentally related indicators appear in the literature, most
of which can be encompassed by the following definition1:

       An environmental indicator is an environmental or environmentally-related variable or estimate,
       or an aggregation of such variables into an index, that is used in some decision-making context:

       •      To show patterns or trends (changes) in the state of the environment (the focus of this
               report),

       •      To show patterns or trends in the human activities that affect, or are affected by, the state
               of the environment,

       •      To show relationships among environmental variables, or

       •      To show relationships between human activities and the state of the environment.

This definition of environmental indicators is purposely very broad to reflect the diversity of assessment
and reporting contexts  in which the term is used.   Thus, the definition includes bom measured  or
observed variables, i.e.,  representative indicators; as well as composite indicators (indices), that aggregate
a number of variables into a single quantity. Representative indicators are measures selected on the basis
of expert opinion or statistical methods to reflect the behavior of one or more variables. Furthermore,
a representative indicator can be either a direct measure of an environmental or environmentally-related
attribute  of interest (e.g., chemical concentrations in effluents, as used in compliance monitoring), or a
surrogate measure (e.g., use of indicator species as surrogate measures of ecosystem integrity), or a more
ecologically-realistic indicator of ecosystem integrity (e.g., community level  measures of biological
condition relative to natural expectations such as the Index of Biotic Integrity). Note that the above
definition of an environmental indicator also includes environmentally-related data (e.g., demographic
data, indicators of human health status), as well as strictly environmental data.
   'larid Sckube. ESID D«ft Caietpeal Affraaetta to At Dntiopmau aid UM tfBmirnmaual Mow* ant SubSct far DuUon-MaUnt, April. 1904.

-------
 Weiut, July 14,1994
FRAMEWORKS FOR DEVELOPING INDICATORS

Many different conceptual frameworks or models of human-environment interactions can be proposed as
bases for the selection, organization, and use of indicators in different policy contexts.  Because the
relationships between human activities and the environment are extremely complex, there is no unique
framework that generates sets of indicators for every purpose, and further, no one framework can
generate a unique set of indicators.

The Organization for Economic Co-operation and Development (OECD) uses a Pressure-State-Response
(PSR) framework2 as  the basis for organizing its reports  on the  State of the  Environment and
environmental performance reviews.  In the basic PSR framework (see Figure 1), human activities exert
pressures on the environment (such as pollution loadings and land use changes), which induce changes
in the state of the environment (such as  ambient levels of pollutants and habitat diversity).  Society
responds to  these changes through environmental and economic policies (such as programs to reduce
impacts to the environment). The latter form a feedback loop to pressures through human activities.  The
OECD has used this PSR framework to develop the core set of indicators now being used hi the OECD
country reviews.

An elaboration of the OECD PSR framework is being developed by EPA as the basis for developing a
system of environmental statistics and indicators, combining it with principles  and techniques  from
environmental assessment and emerging ecosystem  approaches to  decision-making for environmental
management  and development3.  This enhanced conceptual framework adds a category for impacts of
environmental change on human health and welfare and distinguishes between proximate (direct) pressures
and underlying (driving or motivating) pressures.  It also seeks to link the PSR framework explicitly to
society's environmental values, goals and priorities.  Finally, it amis for the incorporation of spatially
referenced  (geographic) information,  organized  on the  basis of ecologically defined land units; the
adoption of sustainability targets; and the multiscaled use of information.

Within the basic PSR framework, three broad types of indicators can be distinguished:
               Indicators of environmental pressures describe the pressures that human activities exert
               on the environment, including the quality and quantity of natural resources.  Indicators
               of environmental  pressures  can be divided into indicators of  proximate pressures
               (pressures directly exerted on the environment, normally expressed in terms of emissions
               or consumption of natural resources) and  indicators of indirect pressures  (background
               indicators reflecting human activities that lead to proximate environmental pressures).

               Indicators of the State of the Environment relate to the quality of the environment and
               the quality and quantity of natural resources. As such, they reflect the ultimate objective
               of environmental policy making. Indicators of environmental conditions should measure
               the state of the environment and changes in that state over time, rather than the pressures
               on it.  In practice, however, the distinction between pressures on the environment and
               the resultant conditions can be ambiguous and the direct measurement of environmental
               conditions can be difficult or very costly. Therefore, the measurement of environmental
  *Hii. PSR dkcuuiao • adcpted from OECD Can Set aJbOcaun far EminanuitJ fafomaHet Jbwmv Environmental Manognpa No. S3 (1993).



      Sduibe. E&tDDntlCaKtpualJfpnadui loAt Dtvtlef^aa a^ VM ef Emin^mlal Mtcaaa tmd Salute, Jar D*cUo»MaH*t. April. 1994.

-------
Figure 1 <§
OECD Pressure - State - Response Framework* 1
H*
• •)
PRESSURES




Human
STATE
RESPONSES
O
— • y
Information 5>
1


Activities
Energy


Transport
Industry
Agriculture
Others


t










Pressures


Resources





State of the Environment



Air
Water
Land
Natural Resources






Information
*"

Societal Responses
(Decisions - Actions)





1 	 :


Society


Public

Private



k 	
Societal Responses (Decisions - Actions)





Sector

Sector



i/3
?
?
' t
re
?
1
*









r
g
><
M
i











3°
;»
y
I
O
w
O
•c
c
§
Adapted from 'OECD Core Set of Indicators for Environmental Performance Reviews". Environmental Monograph No. 83 (1993)

-------
 Weaut, July 14, 1994
                pressures is often used as a substitute for the measurement of environmental conditions.

        •      Indicators of societal responses relate to individual and collective actions to mitigate,
                adapt to, or prevent human-induced damage to the environment and to halt or reverse
                environmental damage already inflicted. Societal responses also include actions for the
                preservation and the conservation of the environment and natural resources.


 EPA's Office of Water uses a six-level continuum of environmental indicators (see Figure 2) to organize
 indicators.  Indicators range from direct measurements of environmental conditions to those that relate
 to agency and state activities. The six types of indicators along the continuum are:
       •    Health and ecological effects                     (State)

       •    Uptake or body burden                          (State)

       •    Ambient levels                                  (State)

       •    Emission or discharge quantities                 (Pressure)

       •    Actions by sources                              (Response)

       •    Agency and state activities                       (Response)


The right-hand column shows the correspondence between the continuum and the OECD PSR framework.
              COMMENTS   ON  THE  PROCESS  OF   SELECTING  ENVIRONMENTAL
INDICATORS

In developing indicators, EPA has come to recognize that environmental indicators need to be explicitly
linked to expressions of society's environmental values, goals and priorities.  Indicators are measurable
quantities that can be related directly or indirectly (via qualitative or quantitative models, e.g., via
indices) to society's environmental values or concerns.  For the State of the Environment, this implies
linking indicators to the specific attributes of ecosystems, human health and welfare that society values
(collectively termed valued environmental attributed).  Indicators of societal pressures and/or responses
need  to be  selected,  among other things,  on the basis of  (current understanding of) risks to  valued
attributes.

Different approaches  can be used to select environmental indicators. On the one hand, indicators for a
particular application can be selected on an ad hoc basis from existing indicators that are already in use.
Such an approach can yield many meaningful and informative indicators;  however, it also limits the range
of possible indicators to those that have been previously developed for other purposes. An alternative
          of vmhud araaniacatal •oributai include •peeMi divetwty, wnlnd hydrolofie function, drinkinc Mter qmtoy, dim* (ability, praduclnfty of Mum.
Mpiplikinu
-------
































Figure 2: A Continuum of Environmental Indicators*












Activity Measures
























Actions by
States/EPA


i

Examples:
revise SIP,
issue permit,
issue grant











— ^














Actions by
Sources







Environmental














Examples:
install control
equipment,
change feedstock



Indirect Indicators







^


Quantified
Pollution
Prevention
Measures

1

Emission/
Discharge
Quantities








^ Ambient
^ Concentrations


i i
t t
i
Risk Estimates
Based on
Emissions
Data
Risk Estimates
Based on
Ambient Data
i 	



Indicators














Direct Indicators






_^. Uptake/
Body Burden


i
i





t
Risk Estimates
Based on Body
Burden Data







f
i
L.


L
T
i
i








Health Effects


Ecological Effects




a
OQ
3
• *
o
e
3
5*
. c
e
_, o
•< -»







i





w
•S-
e
3
§
E
a.
B
0





Adapted from "Measuring Progress to Reach National Goals" EPA Office of Water, Draft September 1993

-------
 Weftit, July 14,1994
 approach is to base indicator selection on (qualitative or quantitative) conceptual model(s) of the issue(s)
 or problem(s) at hand, regardless of current data availability. This approach uses a systematic process
 in which indicator selection is based  on postulated cause-effect linkages between valued environmental
 attributes and the societal and natural factors that potentially affect these attributes.5  While the latter
 approach may expose gaps in existing data, it allows society's environmental values and current scientific
 understanding of environmental linkages to drive indicator selection. The identification of such data gaps
 can drive EPA's indicator research.
CRITERIA FOR SELECTING INDICATORS FOR ASSESSING ENVIRONMENTAL STATUS
AND TRENDS

In  selecting environmental indicators, it  is important to  have clear selection criteria.   Previously
developed criteria are  available from several sources including the Environmental Monitoring  and
Assessment Program (EMAP), the International Joint Commission for the Great Lakes (LTC),  and
Intergovernmental Task Force on Water Quality Monitoring (TTFM). The choice of selection criteria
depends in part on the  intended use for the indicators.  (Note that some indicator selection criteria may
be  mutually exclusive.) The list of criteria suggested here, in Table 1, has been adapted from other
sources (primarily the ITFM criteria).

The selection  criteria are grouped based on  considerations of validity, interpretability, timeliness,
understandability, and cost considerations.  These considerations include the following:

        •      Indicators should be valid measures of the attributes of interest. Validity is defined here
               as a close qualitative or quantitative linkage between the attribute actually of interest
               (e.g., "biological integrity") and  the measurable quantity represented by the indicator.
               Three factors listed  in Table 1 contribute to a close logical link between an indicator and
               the attributes of societal  concern or value.   First, indicators that respond  at  the
               appropriate spatial  and temporal scales  are more likely to be valid measures of an
               attribute of concern.  Second, if the purpose of an indicator is to assess environmental
               status and trends, indicators that respond to cumulative effects of multiple stressors will
               be more representative of overall ecosystem condition than those that are responsive to
               only a few stressors. Third, indicators that are highly correlated with other measures (of
               an specified attribute)  will tend to be representative  of the environmental attribute or
               system being measured. Finally, indicators must be sensitive enough to measure changes
               over a reasonable tune but not so  sensitive that they fluctuate substantially between time
               periods. The signal-to-noise ratio for an indicator is hi pan determined by the data used
               to measure the indicator. Expert knowledge and peer review can be used to assess the
               sensitivity of different indicators.

        •      Indicators should be interpretable hi terms of the end point hi the assessment process.
               They should  be able  to  distinguish  unacceptable  from  acceptable environmental
               conditions.
    5For diuwicn on lh» type of «Hm»ch, MB f or enunple Mamjinj TraMat Waua: Ac gab cfRtjmai MMw Bavinanattl McnJtorini (N
                                                                                              ony
Pra*. \9^*^^h£ca^DtytUfiimt SmUfffor^EitvironmaUal Manuring aidAaamatt Pngnm, U.S. Envin»m«nt«l ProteetMo Apacy, 1991.

-------
Wettat, July 14, 1994
 Table 1:    Criteria for Selecting Indicators
 Criterion
 Validity
 "Social and
 Environmental Relevance
 Appropriate Scale

 Integrates Effects/
 Exposures

 Representative

 Signal-to-Noise Ratio
Explanation
Scientific theory links the indicator to society's environmental values,
goals and concerns.
The indicator responds to changes on an appropriate geographic (e.g.,
national or regional) and temporal (e.g., yearly) scale.
The indicator integrates effects or exposure  over time an space and
responds to the cumulative effects of multiple stressors.  It is broadly
applicable to many stressors and sites.
Changes in the indicator are highly correlated with other measures  of
the attributes of interest.
The  indicator  is  able  to  distinguish  meaningful   differences  in
environmental conditions with an acceptable degree of resolution.
 Interpretability
 * Interpretable

 Comparability

 Timeliness
 Timely/Anticipatory

 Understandability
 * Understandable

 Perceived Relevance to
 the User

 Cost Effectiveness
 Cost Effectiveness

 Minimal Environmental
 Impact

 * Indicates critical criteria
There is a reference condition or benchmark against which to measure
changes and trends.  The indicator can distinguish acceptable conditions
hi a scientifically defensible way.
Can be compared to existing and past measures of conditions to define
trends and variation.
The indicator provides early warning of changes.
Indicator is, or can be transformed into a format that is understandable
to the target audience.
The measured quantity or an index constructed therefrom is seen by the
audience as being important or relevant to their lives.
Information is  available or can be obtained with reasonable cost and
effort. Provides maximum information per unit effort.
Sampling produces minimal environmental impact.

-------
 Wecut, July 14,1994
                Timely indicators that anticipate future changes in the environment are preferred over
                those that are not anticipatory.  To the extent that an indicator does not anticipate future
                conditions, the  indicator with  the least time lag would be  preferred.   The time lag
                depends on both  characteristics of  the indicator and the time lag between the  data
                collection and when the data is available to calculate the indicator.

                Indicators   should  be  understandable by  the  public  and perceived  as relevant.
                Understandability is hi part a characteristic of the indicator and in part a function of how
                the indicator is presented. EPA may need to educate the public on the importance of
                some indicators.  If possible,  indicators should be "attention grabbers" in that they
                communicate to the audience why an  attribute or value is important, e.g., information on
                the  number of fish  is generally  more  interesting to  the  public  than  data  on
                macroinvertebrates in the food  chain.  Keeping data presentations simple,  graphic, and
                consistent  will  help.   When  there is uncertainty  as to  how an indicator will be
                understood, the use of  focus  groups may  help EPA to understand how  the  public
                perceives the indicator and to provide guidance on improvements to the indicator.

                Finally, indicators should be cost effective relative to alternatives, and to the effort and
                expertise to collect the data, if required, and monitor the indicator over tune.
CRITERIA FOR SELECTING EXISTING DATA SETS TO QUANTIFY INDICATORS

Table 2 sets forth proposed criteria for evaluating the usefulness of an existing data set to  assess
environmental conditions and trends in a particular geographic area.

Critical criteria for selecting data sets would include the availability of data on the selected parameters,
appropriate temporal and spatial coverage, documented quality, and accessibility.  Because changes in
the data collection procedures might affect the technical credibility, the magnitude of the estimation error
(and the associated sample size) and the cost, another critical criterion for consideration of a data set is
that minimal standards of technical credibility, estimation precision, and cost can be achieved by either
the present data collection procedures or reasonable modifications.

It is likely that either the sampling procedures or laboratory analysis procedures will change over the time
that a data source is used to quantify an indicator and monitor progress.  These changes will result from
advances in technology and changes in budgets and uses of the data sets over time. The effect of these
changes  can be minimized by using (1) measurements for which changes in technology are likely to
improve the precision but not affect the measurement bias and (2) procedures for which the measurement
bias is relatively insensitive to the magnitude of the collection effort.  To the extent that this cannot be
achieved, a  comparability study can be used to compare the indicator before and  after the change. The
value of both the original and revised indicator can be used for some tune to provide information on how
the two indicator compare.  This same procedure can be used if a entirely new data set is used for the
revised indicator.

-------
Weiut, July 14,1994
 Table 2:    Criteria for Selecting Existing Data Sets to Quantify Indicators
 Criterion

 * Availability of Data
 * Appropriate Temporal
 Coverage
 * Appropriate Spatial
 Coverage
 Documented Quality
 * Accessibility
 Technical Credibility
 Acceptable Estimation
 Error
 Acceptable Cost
   Indicates critical criteria
Explanation

Data set  provides  measurements  of die parameters)  or variable(s)
specified in the indicator.
At a minimum, information should be available for the present and for
future years.  In addition, temporal coverage within reporting cycles
(usually annually) may have gaps but should not exclude data that will
significantly affect the indicator.


Information should be available  on a national (regional) basis for a
national (regional) program, or,  if the information  is compiled  from
local or regional data, the information will need to be aggregated using
scientifically and statistically valid procedures.


The information should be of known quality, i.e., there should be (1)
documented  QA/QC procedures for  the collection,  analysis and
presentation of data, (2) documentation of any  deviations from the
procedures, and  (3) quantitative  information on both sampling and
non-sampling errors.


The information  should  be retrievable and analyzable using existing
data retrieval and analysis procedures.  EPA would  not be prohibited
from using the data due to confidentiality concerns, etc.


The procedures  used to  manage  and analyze the data should  follow
accepted professional practices.   In addition, the  sample and data
collection procedures should not be inconsistent with the use of the
data as a measure of the indicator, as judged by technical experts in
the field who are familiar with the data.   The calculated bias in the
indicator should be insensitive to the magnitude of the data collection
effort  and to political  pressures.   In general, this criterion will
eliminate self-reported data from consideration.


The precision and bias of the indicator should be acceptable given the
desired precision specified by the program.


Cost  of  data collection,  management,   and  analysis are  within
programmatic guidelines.

-------
 Wetut, July 14, 1994
 PROCESS FOR SELECTING INDICATORS AND DATA

 The proposed process  for selecting indicators and data  and filling  information gaps  consists of the
 following four basic steps:

            Step 1.    Identify And Recommend Possible Indicators For Reporting

            Step 2.    Inventory Existing National Data Sets

            Step 3.    Identify and Recommend Information/Data for Selecting Indicators

            Step 4.    Fill Information Gaps

These four steps  are discussed in more detail below. Characteristics of how these four steps interrelate
is provided in Figure 3.

Step 1:     Identify  and Recommend Indicators for Reporting Status and Trends

Step la: Identify possible indicators.  Compile two lists of candidate environmental indicators. The first
list includes indicators  developed on the  basis of conceptual models  specific to the issue of concern,
without any consideration for the availability of data. The second list includes currently used indicators.
In many cases these indicators are meaningful and informative summaries constructed from available data.
Many indicators will appear on both lists.

Step Ib:  Score each indicator, using the criteria hi Table 1, to identify candidate indicators. Review
each indicator hi the two lists and select  a  range of possible indicators for further consideration.
Indicators will be rejected if they do not satisfy all of the critical criteria shown hi Table 1.  Of the
remaining indicators, use a combination of peer review, literature review, and expert knowledge to select
the candidate indicators for further consideration. The availability of data would be used as a criterion
only for selecting between otherwise similar indicators.

Make a preliminary determination (using the descriptions of each indicator and the short  forms for each
data set) of which  data sets are useful for each indicator.   (See Step 3a  below.)  This  will provide
additional information for each indicator on what data may be available.

Step Ic:  Review the specific attributes  to which  indicators should  related, the data  supporting the
indicators, the criteria for selecting indicators, and the data available for each indicator in  the geographic
area of interest (as a secondary criterion) to select  proposed indicators.  The process will  use a
combination of peer review, literature review,  and expert knowledge to select the proposed indicators.

The process of making  a preliminary choice, gathering more information,  and  making a more refined
choice of indicators is  iterative.   Additional  iterations may be necessary to  refine the selection of
indicators and to incorporate new information as it is gathered.
                                               10

-------
WeiUt, June 9, 1994
                            Draft:  Do not cile or quote
Figure 3:      Process for Selecting Indicators and Data and Filling Gaps
' Criteria for
selecting
indicators
^ j
                                         Environmental values,
                                         objectives, and topics
                                       L         \
                          la. Identify possible
                          indicators for reporting
   2a. Inventory existing
     data sets likely
     suitable for use
                         Ib. Score each indicator,
                           identify candidate
                               indicators
                                      \
 2b. Complete preliminary
  characterization of data
sets potentially suitable for
  the candidate indicators
                                         3a. Identify candidate
                                           data sets for each
                                              Indicator
                                     z
                          Ic Review candidate
                        indicators and data sets,
                        select proposed indicators
  \
 2c Complete detailed
    characterization
 of candidate data sets
                                 Criteria for
                             matching data sets
                                and indicators

3b. Identify proposed
data sets for each
proposed indicator, if
possible
Possible
+
— Not possible-^
4a. Document; select
interim indicator;
and fill gaps
i
r
                                                         4b. Use Indicators in reports
                                                   11

-------
 Weittt, July 14,1994
 Step 2:     Inventory  and Describe Existing Data Sets that  may be  Suitable for Quantifying
             Indicators

 This step consists of (1) an ongoing inventory of existing data held by EPA, other federal agencies, and
 other groups that may be suitable for use in reporting on environmental conditions and trends and (2)
 summarizing information about the data sets to facilitate the use of appropriate data sets to measure the
 selected indicators.  This step is broken into three substeps shown in Figure 3 and described below.

 Step 2a:   Inventory  existing  data sets for their  potential suitability for quantifying the candidate
 indicators. Prepare an assessment of existing data sources potentially available for reporting. Continually
 revise the assessment as new information is gathered through discussions with other federal agencies, non-
 government organizations, etc.

 Step 2b:  Complete a preliminary characterization of data sets potentially  suitable for  measuring the
 candidate indicators. After the candidate indicators have been selected, conduct a preliminary assessment
 of each data set that might potentially be used to assess one of the candidate indicators.  To do this,
 prepare a "short form" to screen  each data set.  The "short form" will summarize the most important
 information needed to decide if a data set is potentially appropriate for a selected indicator.  A draft short
 form appears in Appendix A.

 Step 2c:  Complete  a detailed characterization of candidate data sets.  After an initial selection of data
 sets that might be appropriate  for the preliminary  selection of indicators  (see  step  3a), do a more
 extensive examination for  each data set that might be useful for the candidate indicators.  For example,
 a form  provides additional details  beyond that provided by the short form if it can be used to determine
 if a data set (1) provides adequate  data for a selected indicator or (2) provides data that, if augmented or
 modified, can be made adequate.
Step 3:     Identify and Recommend Information/Data for Selected Indicators

Once the indicators to  be used hi reporting assessing environmental  conditions and trends have been
selected, the next step is to select the information/data to be used to quantify the indicators. This requires
examining existing data collection and analysis programs to determine if appropriate information are or
will be available. To the extent that characteristics of the data collection procedures affect the evaluation
of the criteria for selecting indicators - validity, interpretability, sensitivity, timeliness, understandability,
and cost effectiveness - the program should evaluate these criteria hi light of the proposed data set.

The process will be accomplished hi the following two steps:

Step 3a:  Identify candidate data sets for each indicator.  Use the information from the short form,
published EPA documentation, and other sources to identify data sets that might be  appropriate for use
with each indicator. Several data sets might be appropriate for use with an indicator.  If several data sets
could be used, all would be considered unless  one or more were clearly inferior to the others (i.e. being
similar on most criteria but clearly worse on some). Additional information would be collected on all
data sets that were being considered for any of the candidate indicators (step 2c).  In some cases it may
not be possible to identify any data that might be appropriate.

Step 3b:  Identify proposed data sets for each indicator, if possible.  Use additional information to
identify data sets that are either appropriate for use with  each  indicator or,  if not, could be made
acceptable with additional data collection or changes hi procedures.  If several data sets are appropriate


                                               12

-------
Weitet, July 14,1994
for use with an indicator, the best one would generally be chosen.  In some cases it might not be possible
to identify any data that are  appropriate.  For those indicators where the available data are either
inadequate and can be improved or are not available, a data gap exists.
Step 4:     Fill Information Gaps

For indicators that lack adequate data:

        (1)     Document the data gap.

        (2)     Review existing indicators and data to see if some can be used as interim indicators to
               at least provide some information on conditions and trends.

        (3)     Develop  strategies for filling information gaps, including improvements to existing
               programs hi data  collection, data analyses, and information management. Developing
               strategies includes determining if data can be made available by modifying existing data
               management and analysis procedures. For example, this could include the reanalysis of
               existing data or the integration and harmonization of two or more separate data sets.

               a.      If the information can be made available by changes in existing data management
                      or data analysis procedures, develop a strategy for making the needed changes.

               b.      If the information cannot  be made  available by  changes  to  existing data
                      management or data analysis procedures, determine if there are validated test
                      methods, statistical methods, etc. at the levels of precision and bias required:

                      For each  indicator with validated  methods, identify the type of data  required
                      (including statistical design) and design a data collection and analysis program.
                      If feasible, implement the program.

                      For each indicator without validated methods, set up a process to develop these.
                      If needed, set priorities for developing these methods. Once appropriate methods
                      are developed, identify the type of data required (including statistical design and
                      data  analysis) and design a  data collection and analysis program.  If  feasible,
                      implement the program.
                                               13

-------
Wemt, July 14, 1994
                            APPENDIX A




        DRAFT SHORT FORM FOR SCREENING CANDIDATE DATA SETS
                               A-l

-------
Westat, July 14, 1994
                                                        A-2

-------
Weiut, July 14. 1994
                             Data Set Screening ("Short") Form




 Data Set Acronym/Short Name:                        Date Completed: 	/	/
 1. Background and Summary Information



1.1 Full Name of Data Set:



1.2 Sponsoring Agency:



1.3 Contact person:           Name:



                           Address:





                         Telephone:





1.4 Brief summary of data set and the reasons for collecting this data:







1.5 References for additional information









 2. Target Population and Identification of Sampling Units



2.1 Describe the target population / sampling frame:





2.2 Describe the sampling units:





2.3 How were the sampling units covered by the data base selected? (check one)



    [ ]    Probability Based Sampling: Briefly describe the sample design:



    [ ]    Census



    []    Other:  Specify:
                                            A-3

-------
Wettat, July 14, 1994
2.4  Overall response rate:    	%


2.5  What is the geographic coverage of the data in the data set? (check one)

     [ ]   National

     []   Regional:   Specify:

     [ ]   State:       Specify:

     [ ]   Other:      Specify:


2.6  What time period (years) does the data set cover? From 19	to 19	.


2.7  Is the data collection on-going?           Yes [ ]              No [ ]


2.8  Frequency with which the survey or data collection effort is repeated:




 3.  Information Recorded In The Data File

3.1  For  which of the following types of samples (including questionnaires) are measurements/data
     recorded in the data set? (Check all that apply) Describe the samples collected and the measurements
     obtained.

          Sample type           Sample description:            Measurements obtained:

     [ ]    Water

     [ ]    Soil

     []    Air

     []    Food

     [ ]    Bulk chemical

     [ ]    Human tissue or fluid

     [ ]    Questionnaire
          (diary, observation form)
                                             A-4

-------
 Weuat, July 14, 1994
  4. Documentation of Data Collection Procedures           .          .

 4.1 Did the following data collection activities have written procedures (including the use of accepted
     standard methods) and were the procedures documented through a QA/QC program review?

                                           Written.
     Sample/data collection activities        procedures       OA/OC review          Not applicable

     Environmental sampling:                   [ ]                 [ ]                     [ ]

     Biological sampling:                       [ ]                 [ ]                     [ ]

     Field measurements:                       [ ]                 [ ]                     [ ]

     Laboratory preparation and analysis:        [ ]                 [ ]                     [ ]

     Interview/questionnaire/field notes:         []                 []                     []

     Data Entry, editing, and verification:        [ ]                 [ ]                     [ ]
 5. Summary Assessment of Data Quality

S.I For key summary statistics derived from the data, describe the statistic and the associated confidence
    interval or measure of precision:


5.2 Give a general assessment of the data quality. Include comments, issues, or usage guidance relevant
    to using this data, including any potential biases or limitations in the data:
                                             A-5

-------
 Wettt, July 14,1994
  6.  Description of the Available Data File

 6.1  Availability of data:

     Non-aggregated data

     [ ]   Public use. Give cost:

     [ ]   Restricted use:  Specify:

     [ ]   Confidential (not available for
          public use)
Summary Statistics

[ ]  Public use. Give cost:

[]  Restricted use: Specify:

[ ]  Confidential (not available for
    public use)
6.2  In what form are data available?  Mark all that apply.
     Non-aggregated data

     [ ]   Hard copy (for example, Computer
          printouts, Files or log books, Reports,
          Microfilm)

     [ ]   Machine readable form (for example:
          Tape, Diskette, On-line, CD-ROM)
Summary Statistics

[ ]  Hard copy (for example, Computer
    printouts, Files or log books, Reports,
    Microfilm)

[ ]  Machine readable form (for example:
    Tape, Diskette, On-line, CD-ROM)
6.3 Describe available summary statistics:
6.4 On average, how long is the time from field measurement, sample collection, and interviewing until
    data is available to the public?
                                             A-6

-------