PROCESS FOR SELECTING INDICATORS
AND DATA AND FILLING INFORMATION GAPS
Final Report
WESTAT
An Employee-Owned Research Corporation
-------
K:\12oaks\do26\finalrpt.wp5- ah - July 14, 1994
PROCESS FOR SELECTING INDICATORS
AND DATA AND FILLING INFORMATION GAPS
Final Report
Prepared by:
Westat, Inc.
1650 Research Blvd.
Rockville, MD 20850
Prepared for:
U.S. Environmental Protection Agency
401 M Street, S.W.
Washington, DC 20460
This work was conducted under
EPA Contract Number 68-W1-0019
July 14, 1994
-------
TABLE OF CONTENTS
Section Page
INTRODUCTION 1
DEFINITION OF ENVIRONMENTAL INDICATOR 1
FRAMEWORKS FOR DEVELOPING INDICATORS 2
GENERAL COMMENTS ON THE PROCESS OF SELECTING
ENVIRONMENTAL INDICATORS 4
CRITERIA FOR SELECTING INDICATORS FOR ASSESSING
ENVIRONMENTAL STATUS AND TRENDS '. 6
CRITERIA FOR SELECTING EXISTING DATA SETS TO QUANTIFY INDICATORS . . 8
PROCESS FOR SELECTING INDICATORS AND DATA . 10
Step 1. Identify and Recommend Possible Indicators for Reporting Status
and Trends 10
Step 2. Inventory and Describe Existing Data Sets that may be Suitable for
Quantifying Indicators 12
Step 3. Identify and Recommend Information/Data for Selected Indicators 12
Step 4. Fill Information Gaps 13
List of Tables
Table
1 Criteria for Selecting Indicators 7
2 Criteria for Selecting Existing Data Sets to Quantify Indicators 9
List of Figures
Figure
1 OECD Pressure-State-Response Framework 3
2 Continuum of Environmental Indicators 5
3 Process for Selecting Indicators and Data and Filling Gaps 11
11
-------
TABLE OF CONTENTS (continued)
List of Appendices
Appendix Page
A Draft Short Form for Screening Candidate Data Sets A-l
in
-------
WeiUt, July 14,1994
PROCESS FOR SELECTING INDICATORS
AND DATA
AND FILLING INFORMATION GAPS
INTRODUCTION
The purpose of this document is to present a process for selecting indicators and data sets that can be used
to measure the current status of the environment and to show patterns or trends in that status. This
proposed process is directed primarily to technical managers within EPA who have responsibility for the
specification and quantification of indicators.
DEFINITION OF ENVIRONMENTAL INDICATOR
Many definitions of environmental and environmentally related indicators appear in the literature, most
of which can be encompassed by the following definition1:
An environmental indicator is an environmental or environmentally-related variable or estimate,
or an aggregation of such variables into an index, that is used in some decision-making context:
• To show patterns or trends (changes) in the state of the environment (the focus of this
report),
• To show patterns or trends in the human activities that affect, or are affected by, the state
of the environment,
• To show relationships among environmental variables, or
• To show relationships between human activities and the state of the environment.
This definition of environmental indicators is purposely very broad to reflect the diversity of assessment
and reporting contexts in which the term is used. Thus, the definition includes bom measured or
observed variables, i.e., representative indicators; as well as composite indicators (indices), that aggregate
a number of variables into a single quantity. Representative indicators are measures selected on the basis
of expert opinion or statistical methods to reflect the behavior of one or more variables. Furthermore,
a representative indicator can be either a direct measure of an environmental or environmentally-related
attribute of interest (e.g., chemical concentrations in effluents, as used in compliance monitoring), or a
surrogate measure (e.g., use of indicator species as surrogate measures of ecosystem integrity), or a more
ecologically-realistic indicator of ecosystem integrity (e.g., community level measures of biological
condition relative to natural expectations such as the Index of Biotic Integrity). Note that the above
definition of an environmental indicator also includes environmentally-related data (e.g., demographic
data, indicators of human health status), as well as strictly environmental data.
'larid Sckube. ESID D«ft Caietpeal Affraaetta to At Dntiopmau aid UM tfBmirnmaual Mow* ant SubSct far DuUon-MaUnt, April. 1904.
-------
Weiut, July 14,1994
FRAMEWORKS FOR DEVELOPING INDICATORS
Many different conceptual frameworks or models of human-environment interactions can be proposed as
bases for the selection, organization, and use of indicators in different policy contexts. Because the
relationships between human activities and the environment are extremely complex, there is no unique
framework that generates sets of indicators for every purpose, and further, no one framework can
generate a unique set of indicators.
The Organization for Economic Co-operation and Development (OECD) uses a Pressure-State-Response
(PSR) framework2 as the basis for organizing its reports on the State of the Environment and
environmental performance reviews. In the basic PSR framework (see Figure 1), human activities exert
pressures on the environment (such as pollution loadings and land use changes), which induce changes
in the state of the environment (such as ambient levels of pollutants and habitat diversity). Society
responds to these changes through environmental and economic policies (such as programs to reduce
impacts to the environment). The latter form a feedback loop to pressures through human activities. The
OECD has used this PSR framework to develop the core set of indicators now being used hi the OECD
country reviews.
An elaboration of the OECD PSR framework is being developed by EPA as the basis for developing a
system of environmental statistics and indicators, combining it with principles and techniques from
environmental assessment and emerging ecosystem approaches to decision-making for environmental
management and development3. This enhanced conceptual framework adds a category for impacts of
environmental change on human health and welfare and distinguishes between proximate (direct) pressures
and underlying (driving or motivating) pressures. It also seeks to link the PSR framework explicitly to
society's environmental values, goals and priorities. Finally, it amis for the incorporation of spatially
referenced (geographic) information, organized on the basis of ecologically defined land units; the
adoption of sustainability targets; and the multiscaled use of information.
Within the basic PSR framework, three broad types of indicators can be distinguished:
Indicators of environmental pressures describe the pressures that human activities exert
on the environment, including the quality and quantity of natural resources. Indicators
of environmental pressures can be divided into indicators of proximate pressures
(pressures directly exerted on the environment, normally expressed in terms of emissions
or consumption of natural resources) and indicators of indirect pressures (background
indicators reflecting human activities that lead to proximate environmental pressures).
Indicators of the State of the Environment relate to the quality of the environment and
the quality and quantity of natural resources. As such, they reflect the ultimate objective
of environmental policy making. Indicators of environmental conditions should measure
the state of the environment and changes in that state over time, rather than the pressures
on it. In practice, however, the distinction between pressures on the environment and
the resultant conditions can be ambiguous and the direct measurement of environmental
conditions can be difficult or very costly. Therefore, the measurement of environmental
*Hii. PSR dkcuuiao • adcpted from OECD Can Set aJbOcaun far EminanuitJ fafomaHet Jbwmv Environmental Manognpa No. S3 (1993).
Sduibe. E&tDDntlCaKtpualJfpnadui loAt Dtvtlef^aa a^ VM ef Emin^mlal Mtcaaa tmd Salute, Jar D*cUo»MaH*t. April. 1994.
-------
Figure 1 <§
OECD Pressure - State - Response Framework* 1
H*
• •)
PRESSURES
Human
STATE
RESPONSES
O
— • y
Information 5>
1
Activities
Energy
Transport
Industry
Agriculture
Others
t
Pressures
Resources
State of the Environment
Air
Water
Land
Natural Resources
Information
*"
Societal Responses
(Decisions - Actions)
1 :
Society
Public
Private
k
Societal Responses (Decisions - Actions)
Sector
Sector
i/3
?
?
' t
re
?
1
*
r
g
><
M
i
3°
;»
y
I
O
w
O
•c
c
§
Adapted from 'OECD Core Set of Indicators for Environmental Performance Reviews". Environmental Monograph No. 83 (1993)
-------
Weaut, July 14, 1994
pressures is often used as a substitute for the measurement of environmental conditions.
• Indicators of societal responses relate to individual and collective actions to mitigate,
adapt to, or prevent human-induced damage to the environment and to halt or reverse
environmental damage already inflicted. Societal responses also include actions for the
preservation and the conservation of the environment and natural resources.
EPA's Office of Water uses a six-level continuum of environmental indicators (see Figure 2) to organize
indicators. Indicators range from direct measurements of environmental conditions to those that relate
to agency and state activities. The six types of indicators along the continuum are:
• Health and ecological effects (State)
• Uptake or body burden (State)
• Ambient levels (State)
• Emission or discharge quantities (Pressure)
• Actions by sources (Response)
• Agency and state activities (Response)
The right-hand column shows the correspondence between the continuum and the OECD PSR framework.
COMMENTS ON THE PROCESS OF SELECTING ENVIRONMENTAL
INDICATORS
In developing indicators, EPA has come to recognize that environmental indicators need to be explicitly
linked to expressions of society's environmental values, goals and priorities. Indicators are measurable
quantities that can be related directly or indirectly (via qualitative or quantitative models, e.g., via
indices) to society's environmental values or concerns. For the State of the Environment, this implies
linking indicators to the specific attributes of ecosystems, human health and welfare that society values
(collectively termed valued environmental attributed). Indicators of societal pressures and/or responses
need to be selected, among other things, on the basis of (current understanding of) risks to valued
attributes.
Different approaches can be used to select environmental indicators. On the one hand, indicators for a
particular application can be selected on an ad hoc basis from existing indicators that are already in use.
Such an approach can yield many meaningful and informative indicators; however, it also limits the range
of possible indicators to those that have been previously developed for other purposes. An alternative
of vmhud araaniacatal •oributai include •peeMi divetwty, wnlnd hydrolofie function, drinkinc Mter qmtoy, dim* (ability, praduclnfty of Mum.
Mpiplikinu
-------
Figure 2: A Continuum of Environmental Indicators*
Activity Measures
Actions by
States/EPA
i
Examples:
revise SIP,
issue permit,
issue grant
— ^
Actions by
Sources
Environmental
Examples:
install control
equipment,
change feedstock
Indirect Indicators
^
Quantified
Pollution
Prevention
Measures
1
Emission/
Discharge
Quantities
^ Ambient
^ Concentrations
i i
t t
i
Risk Estimates
Based on
Emissions
Data
Risk Estimates
Based on
Ambient Data
i
Indicators
Direct Indicators
_^. Uptake/
Body Burden
i
i
t
Risk Estimates
Based on Body
Burden Data
f
i
L.
L
T
i
i
Health Effects
Ecological Effects
a
OQ
3
• *
o
e
3
5*
. c
e
_, o
•< -»
i
w
•S-
e
3
§
E
a.
B
0
Adapted from "Measuring Progress to Reach National Goals" EPA Office of Water, Draft September 1993
-------
Weftit, July 14,1994
approach is to base indicator selection on (qualitative or quantitative) conceptual model(s) of the issue(s)
or problem(s) at hand, regardless of current data availability. This approach uses a systematic process
in which indicator selection is based on postulated cause-effect linkages between valued environmental
attributes and the societal and natural factors that potentially affect these attributes.5 While the latter
approach may expose gaps in existing data, it allows society's environmental values and current scientific
understanding of environmental linkages to drive indicator selection. The identification of such data gaps
can drive EPA's indicator research.
CRITERIA FOR SELECTING INDICATORS FOR ASSESSING ENVIRONMENTAL STATUS
AND TRENDS
In selecting environmental indicators, it is important to have clear selection criteria. Previously
developed criteria are available from several sources including the Environmental Monitoring and
Assessment Program (EMAP), the International Joint Commission for the Great Lakes (LTC), and
Intergovernmental Task Force on Water Quality Monitoring (TTFM). The choice of selection criteria
depends in part on the intended use for the indicators. (Note that some indicator selection criteria may
be mutually exclusive.) The list of criteria suggested here, in Table 1, has been adapted from other
sources (primarily the ITFM criteria).
The selection criteria are grouped based on considerations of validity, interpretability, timeliness,
understandability, and cost considerations. These considerations include the following:
• Indicators should be valid measures of the attributes of interest. Validity is defined here
as a close qualitative or quantitative linkage between the attribute actually of interest
(e.g., "biological integrity") and the measurable quantity represented by the indicator.
Three factors listed in Table 1 contribute to a close logical link between an indicator and
the attributes of societal concern or value. First, indicators that respond at the
appropriate spatial and temporal scales are more likely to be valid measures of an
attribute of concern. Second, if the purpose of an indicator is to assess environmental
status and trends, indicators that respond to cumulative effects of multiple stressors will
be more representative of overall ecosystem condition than those that are responsive to
only a few stressors. Third, indicators that are highly correlated with other measures (of
an specified attribute) will tend to be representative of the environmental attribute or
system being measured. Finally, indicators must be sensitive enough to measure changes
over a reasonable tune but not so sensitive that they fluctuate substantially between time
periods. The signal-to-noise ratio for an indicator is hi pan determined by the data used
to measure the indicator. Expert knowledge and peer review can be used to assess the
sensitivity of different indicators.
• Indicators should be interpretable hi terms of the end point hi the assessment process.
They should be able to distinguish unacceptable from acceptable environmental
conditions.
5For diuwicn on lh» type of «Hm»ch, MB f or enunple Mamjinj TraMat Waua: Ac gab cfRtjmai MMw Bavinanattl McnJtorini (N
ony
Pra*. \9^*^^h£ca^DtytUfiimt SmUfffor^EitvironmaUal Manuring aidAaamatt Pngnm, U.S. Envin»m«nt«l ProteetMo Apacy, 1991.
-------
Wettat, July 14, 1994
Table 1: Criteria for Selecting Indicators
Criterion
Validity
"Social and
Environmental Relevance
Appropriate Scale
Integrates Effects/
Exposures
Representative
Signal-to-Noise Ratio
Explanation
Scientific theory links the indicator to society's environmental values,
goals and concerns.
The indicator responds to changes on an appropriate geographic (e.g.,
national or regional) and temporal (e.g., yearly) scale.
The indicator integrates effects or exposure over time an space and
responds to the cumulative effects of multiple stressors. It is broadly
applicable to many stressors and sites.
Changes in the indicator are highly correlated with other measures of
the attributes of interest.
The indicator is able to distinguish meaningful differences in
environmental conditions with an acceptable degree of resolution.
Interpretability
* Interpretable
Comparability
Timeliness
Timely/Anticipatory
Understandability
* Understandable
Perceived Relevance to
the User
Cost Effectiveness
Cost Effectiveness
Minimal Environmental
Impact
* Indicates critical criteria
There is a reference condition or benchmark against which to measure
changes and trends. The indicator can distinguish acceptable conditions
hi a scientifically defensible way.
Can be compared to existing and past measures of conditions to define
trends and variation.
The indicator provides early warning of changes.
Indicator is, or can be transformed into a format that is understandable
to the target audience.
The measured quantity or an index constructed therefrom is seen by the
audience as being important or relevant to their lives.
Information is available or can be obtained with reasonable cost and
effort. Provides maximum information per unit effort.
Sampling produces minimal environmental impact.
-------
Wecut, July 14,1994
Timely indicators that anticipate future changes in the environment are preferred over
those that are not anticipatory. To the extent that an indicator does not anticipate future
conditions, the indicator with the least time lag would be preferred. The time lag
depends on both characteristics of the indicator and the time lag between the data
collection and when the data is available to calculate the indicator.
Indicators should be understandable by the public and perceived as relevant.
Understandability is hi part a characteristic of the indicator and in part a function of how
the indicator is presented. EPA may need to educate the public on the importance of
some indicators. If possible, indicators should be "attention grabbers" in that they
communicate to the audience why an attribute or value is important, e.g., information on
the number of fish is generally more interesting to the public than data on
macroinvertebrates in the food chain. Keeping data presentations simple, graphic, and
consistent will help. When there is uncertainty as to how an indicator will be
understood, the use of focus groups may help EPA to understand how the public
perceives the indicator and to provide guidance on improvements to the indicator.
Finally, indicators should be cost effective relative to alternatives, and to the effort and
expertise to collect the data, if required, and monitor the indicator over tune.
CRITERIA FOR SELECTING EXISTING DATA SETS TO QUANTIFY INDICATORS
Table 2 sets forth proposed criteria for evaluating the usefulness of an existing data set to assess
environmental conditions and trends in a particular geographic area.
Critical criteria for selecting data sets would include the availability of data on the selected parameters,
appropriate temporal and spatial coverage, documented quality, and accessibility. Because changes in
the data collection procedures might affect the technical credibility, the magnitude of the estimation error
(and the associated sample size) and the cost, another critical criterion for consideration of a data set is
that minimal standards of technical credibility, estimation precision, and cost can be achieved by either
the present data collection procedures or reasonable modifications.
It is likely that either the sampling procedures or laboratory analysis procedures will change over the time
that a data source is used to quantify an indicator and monitor progress. These changes will result from
advances in technology and changes in budgets and uses of the data sets over time. The effect of these
changes can be minimized by using (1) measurements for which changes in technology are likely to
improve the precision but not affect the measurement bias and (2) procedures for which the measurement
bias is relatively insensitive to the magnitude of the collection effort. To the extent that this cannot be
achieved, a comparability study can be used to compare the indicator before and after the change. The
value of both the original and revised indicator can be used for some tune to provide information on how
the two indicator compare. This same procedure can be used if a entirely new data set is used for the
revised indicator.
-------
Weiut, July 14,1994
Table 2: Criteria for Selecting Existing Data Sets to Quantify Indicators
Criterion
* Availability of Data
* Appropriate Temporal
Coverage
* Appropriate Spatial
Coverage
Documented Quality
* Accessibility
Technical Credibility
Acceptable Estimation
Error
Acceptable Cost
Indicates critical criteria
Explanation
Data set provides measurements of die parameters) or variable(s)
specified in the indicator.
At a minimum, information should be available for the present and for
future years. In addition, temporal coverage within reporting cycles
(usually annually) may have gaps but should not exclude data that will
significantly affect the indicator.
Information should be available on a national (regional) basis for a
national (regional) program, or, if the information is compiled from
local or regional data, the information will need to be aggregated using
scientifically and statistically valid procedures.
The information should be of known quality, i.e., there should be (1)
documented QA/QC procedures for the collection, analysis and
presentation of data, (2) documentation of any deviations from the
procedures, and (3) quantitative information on both sampling and
non-sampling errors.
The information should be retrievable and analyzable using existing
data retrieval and analysis procedures. EPA would not be prohibited
from using the data due to confidentiality concerns, etc.
The procedures used to manage and analyze the data should follow
accepted professional practices. In addition, the sample and data
collection procedures should not be inconsistent with the use of the
data as a measure of the indicator, as judged by technical experts in
the field who are familiar with the data. The calculated bias in the
indicator should be insensitive to the magnitude of the data collection
effort and to political pressures. In general, this criterion will
eliminate self-reported data from consideration.
The precision and bias of the indicator should be acceptable given the
desired precision specified by the program.
Cost of data collection, management, and analysis are within
programmatic guidelines.
-------
Wetut, July 14, 1994
PROCESS FOR SELECTING INDICATORS AND DATA
The proposed process for selecting indicators and data and filling information gaps consists of the
following four basic steps:
Step 1. Identify And Recommend Possible Indicators For Reporting
Step 2. Inventory Existing National Data Sets
Step 3. Identify and Recommend Information/Data for Selecting Indicators
Step 4. Fill Information Gaps
These four steps are discussed in more detail below. Characteristics of how these four steps interrelate
is provided in Figure 3.
Step 1: Identify and Recommend Indicators for Reporting Status and Trends
Step la: Identify possible indicators. Compile two lists of candidate environmental indicators. The first
list includes indicators developed on the basis of conceptual models specific to the issue of concern,
without any consideration for the availability of data. The second list includes currently used indicators.
In many cases these indicators are meaningful and informative summaries constructed from available data.
Many indicators will appear on both lists.
Step Ib: Score each indicator, using the criteria hi Table 1, to identify candidate indicators. Review
each indicator hi the two lists and select a range of possible indicators for further consideration.
Indicators will be rejected if they do not satisfy all of the critical criteria shown hi Table 1. Of the
remaining indicators, use a combination of peer review, literature review, and expert knowledge to select
the candidate indicators for further consideration. The availability of data would be used as a criterion
only for selecting between otherwise similar indicators.
Make a preliminary determination (using the descriptions of each indicator and the short forms for each
data set) of which data sets are useful for each indicator. (See Step 3a below.) This will provide
additional information for each indicator on what data may be available.
Step Ic: Review the specific attributes to which indicators should related, the data supporting the
indicators, the criteria for selecting indicators, and the data available for each indicator in the geographic
area of interest (as a secondary criterion) to select proposed indicators. The process will use a
combination of peer review, literature review, and expert knowledge to select the proposed indicators.
The process of making a preliminary choice, gathering more information, and making a more refined
choice of indicators is iterative. Additional iterations may be necessary to refine the selection of
indicators and to incorporate new information as it is gathered.
10
-------
WeiUt, June 9, 1994
Draft: Do not cile or quote
Figure 3: Process for Selecting Indicators and Data and Filling Gaps
' Criteria for
selecting
indicators
^ j
Environmental values,
objectives, and topics
L \
la. Identify possible
indicators for reporting
2a. Inventory existing
data sets likely
suitable for use
Ib. Score each indicator,
identify candidate
indicators
\
2b. Complete preliminary
characterization of data
sets potentially suitable for
the candidate indicators
3a. Identify candidate
data sets for each
Indicator
z
Ic Review candidate
indicators and data sets,
select proposed indicators
\
2c Complete detailed
characterization
of candidate data sets
Criteria for
matching data sets
and indicators
3b. Identify proposed
data sets for each
proposed indicator, if
possible
Possible
+
— Not possible-^
4a. Document; select
interim indicator;
and fill gaps
i
r
4b. Use Indicators in reports
11
-------
Weittt, July 14,1994
Step 2: Inventory and Describe Existing Data Sets that may be Suitable for Quantifying
Indicators
This step consists of (1) an ongoing inventory of existing data held by EPA, other federal agencies, and
other groups that may be suitable for use in reporting on environmental conditions and trends and (2)
summarizing information about the data sets to facilitate the use of appropriate data sets to measure the
selected indicators. This step is broken into three substeps shown in Figure 3 and described below.
Step 2a: Inventory existing data sets for their potential suitability for quantifying the candidate
indicators. Prepare an assessment of existing data sources potentially available for reporting. Continually
revise the assessment as new information is gathered through discussions with other federal agencies, non-
government organizations, etc.
Step 2b: Complete a preliminary characterization of data sets potentially suitable for measuring the
candidate indicators. After the candidate indicators have been selected, conduct a preliminary assessment
of each data set that might potentially be used to assess one of the candidate indicators. To do this,
prepare a "short form" to screen each data set. The "short form" will summarize the most important
information needed to decide if a data set is potentially appropriate for a selected indicator. A draft short
form appears in Appendix A.
Step 2c: Complete a detailed characterization of candidate data sets. After an initial selection of data
sets that might be appropriate for the preliminary selection of indicators (see step 3a), do a more
extensive examination for each data set that might be useful for the candidate indicators. For example,
a form provides additional details beyond that provided by the short form if it can be used to determine
if a data set (1) provides adequate data for a selected indicator or (2) provides data that, if augmented or
modified, can be made adequate.
Step 3: Identify and Recommend Information/Data for Selected Indicators
Once the indicators to be used hi reporting assessing environmental conditions and trends have been
selected, the next step is to select the information/data to be used to quantify the indicators. This requires
examining existing data collection and analysis programs to determine if appropriate information are or
will be available. To the extent that characteristics of the data collection procedures affect the evaluation
of the criteria for selecting indicators - validity, interpretability, sensitivity, timeliness, understandability,
and cost effectiveness - the program should evaluate these criteria hi light of the proposed data set.
The process will be accomplished hi the following two steps:
Step 3a: Identify candidate data sets for each indicator. Use the information from the short form,
published EPA documentation, and other sources to identify data sets that might be appropriate for use
with each indicator. Several data sets might be appropriate for use with an indicator. If several data sets
could be used, all would be considered unless one or more were clearly inferior to the others (i.e. being
similar on most criteria but clearly worse on some). Additional information would be collected on all
data sets that were being considered for any of the candidate indicators (step 2c). In some cases it may
not be possible to identify any data that might be appropriate.
Step 3b: Identify proposed data sets for each indicator, if possible. Use additional information to
identify data sets that are either appropriate for use with each indicator or, if not, could be made
acceptable with additional data collection or changes hi procedures. If several data sets are appropriate
12
-------
Weitet, July 14,1994
for use with an indicator, the best one would generally be chosen. In some cases it might not be possible
to identify any data that are appropriate. For those indicators where the available data are either
inadequate and can be improved or are not available, a data gap exists.
Step 4: Fill Information Gaps
For indicators that lack adequate data:
(1) Document the data gap.
(2) Review existing indicators and data to see if some can be used as interim indicators to
at least provide some information on conditions and trends.
(3) Develop strategies for filling information gaps, including improvements to existing
programs hi data collection, data analyses, and information management. Developing
strategies includes determining if data can be made available by modifying existing data
management and analysis procedures. For example, this could include the reanalysis of
existing data or the integration and harmonization of two or more separate data sets.
a. If the information can be made available by changes in existing data management
or data analysis procedures, develop a strategy for making the needed changes.
b. If the information cannot be made available by changes to existing data
management or data analysis procedures, determine if there are validated test
methods, statistical methods, etc. at the levels of precision and bias required:
For each indicator with validated methods, identify the type of data required
(including statistical design) and design a data collection and analysis program.
If feasible, implement the program.
For each indicator without validated methods, set up a process to develop these.
If needed, set priorities for developing these methods. Once appropriate methods
are developed, identify the type of data required (including statistical design and
data analysis) and design a data collection and analysis program. If feasible,
implement the program.
13
-------
Wemt, July 14, 1994
APPENDIX A
DRAFT SHORT FORM FOR SCREENING CANDIDATE DATA SETS
A-l
-------
Westat, July 14, 1994
A-2
-------
Weiut, July 14. 1994
Data Set Screening ("Short") Form
Data Set Acronym/Short Name: Date Completed: / /
1. Background and Summary Information
1.1 Full Name of Data Set:
1.2 Sponsoring Agency:
1.3 Contact person: Name:
Address:
Telephone:
1.4 Brief summary of data set and the reasons for collecting this data:
1.5 References for additional information
2. Target Population and Identification of Sampling Units
2.1 Describe the target population / sampling frame:
2.2 Describe the sampling units:
2.3 How were the sampling units covered by the data base selected? (check one)
[ ] Probability Based Sampling: Briefly describe the sample design:
[ ] Census
[] Other: Specify:
A-3
-------
Wettat, July 14, 1994
2.4 Overall response rate: %
2.5 What is the geographic coverage of the data in the data set? (check one)
[ ] National
[] Regional: Specify:
[ ] State: Specify:
[ ] Other: Specify:
2.6 What time period (years) does the data set cover? From 19 to 19 .
2.7 Is the data collection on-going? Yes [ ] No [ ]
2.8 Frequency with which the survey or data collection effort is repeated:
3. Information Recorded In The Data File
3.1 For which of the following types of samples (including questionnaires) are measurements/data
recorded in the data set? (Check all that apply) Describe the samples collected and the measurements
obtained.
Sample type Sample description: Measurements obtained:
[ ] Water
[ ] Soil
[] Air
[] Food
[ ] Bulk chemical
[ ] Human tissue or fluid
[ ] Questionnaire
(diary, observation form)
A-4
-------
Weuat, July 14, 1994
4. Documentation of Data Collection Procedures . .
4.1 Did the following data collection activities have written procedures (including the use of accepted
standard methods) and were the procedures documented through a QA/QC program review?
Written.
Sample/data collection activities procedures OA/OC review Not applicable
Environmental sampling: [ ] [ ] [ ]
Biological sampling: [ ] [ ] [ ]
Field measurements: [ ] [ ] [ ]
Laboratory preparation and analysis: [ ] [ ] [ ]
Interview/questionnaire/field notes: [] [] []
Data Entry, editing, and verification: [ ] [ ] [ ]
5. Summary Assessment of Data Quality
S.I For key summary statistics derived from the data, describe the statistic and the associated confidence
interval or measure of precision:
5.2 Give a general assessment of the data quality. Include comments, issues, or usage guidance relevant
to using this data, including any potential biases or limitations in the data:
A-5
-------
Wettt, July 14,1994
6. Description of the Available Data File
6.1 Availability of data:
Non-aggregated data
[ ] Public use. Give cost:
[ ] Restricted use: Specify:
[ ] Confidential (not available for
public use)
Summary Statistics
[ ] Public use. Give cost:
[] Restricted use: Specify:
[ ] Confidential (not available for
public use)
6.2 In what form are data available? Mark all that apply.
Non-aggregated data
[ ] Hard copy (for example, Computer
printouts, Files or log books, Reports,
Microfilm)
[ ] Machine readable form (for example:
Tape, Diskette, On-line, CD-ROM)
Summary Statistics
[ ] Hard copy (for example, Computer
printouts, Files or log books, Reports,
Microfilm)
[ ] Machine readable form (for example:
Tape, Diskette, On-line, CD-ROM)
6.3 Describe available summary statistics:
6.4 On average, how long is the time from field measurement, sample collection, and interviewing until
data is available to the public?
A-6
------- |