tv.uil C'.-iu1 ULO'i
PonnsAunia Avenue NW
-?-'...20
PROCESS FOR SELECTING IND
SUPPORTING D
Second Editioi
jpared by:
Erflipronmental Protection Agency
Data Quality Action Team
401 M Street, SW
Washington, DC 20460
U S. EPA Headquarters Library
Mail code 3SW- 3404
1200 Pennsylvania Avenue NW
Washington DC 20460
EPA Contract Number 68-W4-0031, D04
SAIC Project #01-0833-07-4977-030
-------
-------
TABLE OF CONTENTS
t.O INTRODUCTION I
2.0 DEFINITION OF ENVIRONMENTAL INDICATOR . . ; 2
3.0 FRAMEWORKS FOR DEVELOPING INDICATORS 5
4.0 PROCESS FOR SELECTING ENVIRONMENTAL INDICATORS 10
5.0 CRITERIA FOR SELECTING ENVIRONMENTAL INDICATORS 22
6.0 CRITERIA FOR SELECTING EXISTING DATA SETS TO
SUPPORT INDICATORS 32
LIST OF FIGURES
Figure 2-1. The Information Pyramid (adapted from State Environmental Goals and
Indicators Project, 1995 and Hammond, Adriaanse, et al., 1995) 4
Figure 3-1. OECD Pressure-State-Response Framework (Adapted from "OECD
Core Set of Indicators for Environmental Performance Review,"
Environmental Monograph No. 83 (1993). 6
Figure 4-1. Process for Selecting Indicators 13
Figure A-1. Pressure-State-Response/Effects (PSR/E) Framework (adapted from
USEPA, 1995) A-2
LIST OF TABLES
Table 5-1. Suggested Evaluation Criteria for Selecting Indicators 23
Table 5-2. Example Development and Application of Suggested Evaluation Criteria . . 27
Table 5-3. Example Approach for Using Weighted Criteria for Evaluating Possible
Indicators 29
Table 6-1. Criteria for Selecting Existing Data Sets to Quantify Indicators 34
Table A-l. Case Study of Pressure, State, Response/Effects Framework and
Subcategories (adapted from USEPA, 1995) .......' A-3
Revised Draft
May 1996
-------
Process for Selecting Indicators
Exhibit 4-1.
Exhibit 4-2.
Exhibit 4-3.
LIST OF EXHIBITS
Example Application of Task l(a) 15
Example Application of Task l(b) 16
State Environmental Goals and Indicators Project Qualification
Standards" -.- 20
Exhibit 4-4. State Environmental Goals and Indicators Project Classification
Scheme23 21
Appendix A-l.
Appendix B-l.
Appendix C-l.
LIST OF APPENDICES
Pressure-State-Response/Effects Framework A-l
Draft Short Form for Screening Candidate Data Sets B-t
Selection Criteria Used in Other Environmental Indicator Projects . C-l
Revised Draft
11
May 1996
-------
1.0 INTRODUCTION
Offices within the U.S. Environmental Protection Agency (EPA) have been developing
processes for selecting environmental indicators and supporting data. In early 1994. the EPA
Data Quality Action Team (Data QAT), comprising representatives from many EPA.offices,
prepared the first edition of this document as a tool for selecting indicators. The earlier edition
has now been updated and revised to reflect lessons learned in earlier work.
The purpose of this document is to present a process for selecting indicators and data sets
that can be used to measure the current status of the environment and to show patterns or trends
in that status. This proposed process, which is endorsed by the Data QAT (which includes
members from virtually all of the EPA programs involved in indicator selection), is directed
primarily to technical managers within EPA who are responsible for specifying and' quantifying
indicators.
This document is organized as follows:
• Section 2.0, Definition of Environmental Indicator—Provides background on
definitions and uses of environmental indicators
• Section 3.0, Frameworks for Developing Indicators—Describes a commonly used
framework for the organization and presentation of environmental indicators
• Section 4.0, Process for Selecting Environmental Indicators—Recommends steps for
Indicator Teams and stakeholder groups to select the most appropriate indicators for
particular projects
• Section 5.0, Criteria for Selecting Environmental Indicators—Describes the process
of determining and applying indicator selection criteria as a means to focus the
• selection process by evaluating candidate indicators
• Section 6.0, Criteria for Selecting Existing Data Sets to Quantify Indicators-
Identifies proposed criteria for evaluating the usefulness of an existing data set to
support environmental indicators.
Revised Draft 1 May 1996
-------
2.0 DEFINITION OF ENVIRONMENTAL INDICATOR
An indicator is most commonly understood as a sign or signal that relays a complex
message in a simplified manner. Environmental indicators describe, analyze, summarize, arid
present scientifically based information on environmental conditions, trends, and their
significance.1 Both direct measures of environmental attributes of interest (e.g., health and
ecological effects) and indirect measures (e.g., emission/cischarge quantities) can serve as
indicators. Environmental indicators are usually presented statistically or graphically to simplify
complex environmental issues by I) quantifying information to highlight its significance and
2) presenting the information in a useful format for communicating ideas and trends related to
the issue.2
The definition of indicators is dynamic and flexible and is influenced by project-specific
factors, including purpose, scope, and target audience. Many definitions of environmental and
environmentally related indicators appear in the literature. Table 2-1 presents selected definitions
[table to be added].
Indicators are developed to quantify and simplify lar.»e amounts of information, thereby
making it more useful for the audience. An indicator can be used individually (e.g., ambient
pollution concentrations), but is more commonly used with other indicators to tell (a more
complete story. In some instances, several individual indicators are grouped under a common
theme .(e.g., the theme Toxics in the Chesapeake Bay may contain several indicators: Toxic
Release Inventory summaries on loadings and releases, trends of contaminants in bottom sediment
and ambient surface water, pesticide use by county). Indicators can be presented individually in
a slide or fact sheet or compiled in an environmental bulletin, multimedia presentation, or other
more comprehensive presentation. Sometimes two or more indicators are presented together on
a single indicator graph to illustrate possible associations among several related pieces of
'State Environmental Goals and Indicators Project. 1995. Prospective Indicators for State Use in Performance
Agreements. Florida Center for Public Management, Florida State University.
:Hammond. A.. A. Adriaanse, E. Rodenburg, D. Bryant. R. Woodward. 1995. Environmental Indicators: A
Systematic Approach to Measuring and Reporting on Environmental Policy Performance in the Context of
Sustainable Development. Washington. D.C.: World Resources Institute.
Revised Draft
May 1996
-------
Definition of Environmental Indicaior
information (e.g.. municipal wastewater treatment plant loadings and population change). Data
can also be aggregated into an index, which is then presented as an indicator. For example, the
Pollutant Standard Index aggregates individual measures of air quality into a single indicator that
rates daily air quality as good, unhealthful. or hazardous. The scope of the message
communicated by an indicator can be at any of several different levels—community, sectoral,
national, or international.
Typically, environmental indicators are used either in a decision-making context or as a
tool for public education/outreach to achieve the following purposes:
• Show patterns or trends (changes) in the state of the environment (such indicators are
the focus of this report)
• Show patterns or trends in the human activities that affect or are affected by the state
of the environment
• Show relationships among environmental variables
• Show relationships between human activities and the state of the environment
• Provide a benchmark against which to measure progress toward a particular goal
• Communicate a message, theme, or story clearly, succinctly, and accurately
• Motivate the readers to change behavior
• Correct misperceptions.
Because indicators are user-driven and are characterized by the quantification and
simplification of important information, a specific relationship exists among indicators, the
audience, and the level of data. The information pyramid, shown in Figure 2-1, contains three
levels of data for three different target audiences.3 The base of the pyramid is primary data
derived from monitoring and data analysis. Primary data are most useful for scientists or for an
audience with a more technical background. The next layer is analyzed data, which decision
makers frequently use to quickly assess and evaluate trends, such as progress toward achieving
'State Environmental Goals and Indicators Project, op. cit.
Revised Draft . 3 May 19%
-------
Process for Selecting Indicators
policy goais. Proceeding upward, the next layer represents indicators, which are the most
aggregated forms of data. Very condensed amounts of highly aggregated data are useful for
larger audiences with a less technical background, such as the general public.
Figure 2-1. The Information Pyramid
(adapted from State Environmental Goals and Indicators Project, 1995 and
Hammond, Adriaanse, et ah, 1995)
Revised Draft
May 1996
-------
3.0 FRAMEWORKS FOR DEVELOPING INDICATORS
This section will present a framework commonly used for the development of
environmental indicators, known as' the Pressure-State-Response (PSR) framework. Used by
various environmental organizations for environmental indicator projects, this framework serves
to convey environmental information in a coherent way. A detailed description of the PSR
framework and typical characteristics of indicators developed under each component is provided
below. Readers familiar with the PSR framework may continue on to Section 4.0 for information
on the process of indicator selection.
Environmental indicators synthesize complex, scientific information in a simplified and
understandable manner. Although the content and presentation style of indicators -may vary
depending on such factors as the intended use, target audience, and message (or theme) of the
indicators, most indicators are derived from an extensive information base. Effective frameworks
are needed to provide context for the indicator and to structure the diverse environmental
information so that it is relevant, interpretable, accessible, and intelligible to the target audience
(e.g., decision makers, general public).4 Developing indicators within a specific framework
promotes effective information collection, integration, and interpretation (e.g., linking
environment-related data to policy and management actions or needs), while also revealing
potential data gaps and providing the impetus for future data collection efforts.5
A variety of conceptual frameworks or models of human-environment interactions can
serve as the basis for selecting, organizing, and using indicators in different policy contexts.
Because the relationships between human activities and the environment are extremely complex,
no one framework may meet the needs of every indicator project. The PSR framework, adopted
by the Organization for Economic Co-operation and Development (OECD) as the basis for
organizing its State of the Environment reports and environmental performance reviews (OECD,
'Hammond, op. cit.
-'United States Environmental Protection Agency. 1995. A Conceptual Framework To Support Development And
Use Of Environmental Information In Decision-Making. EPA 239-R-95-012.
i
Revised Draft 5 MaX 1996
-------
Process for Selecting Indicators
1993). is widely used, however, for organizing environmental indicators and supporting
information.
The PSR framework6 provides a valuable means for relating and .integrating
environmental information necessary for developing effective indicators that are capable of telling
a story or conveying a discrete message. The basic PSR framework, given in Figure 3-1,
establishes a causal relationship among human activities, the state of the environment, and
society's response. Human activities exert pressures on the environment (e.g., pollution loadings
and land use changes) and induce changes in the state of die environment (e.g., ambient levels
of pollutants and habitat diversity). Society responds to these changes by addressing the
pressures through environmental and economic policies (e.|(., programs to reduce impacts to the
environment).
PRESSL
1
Human
«
Energy
Tnrwport
Industry
Agricuttur*
Othwm
A
RES |
•
N-
i^W^^^^V^^P J
V
\
STATE
Informttian
•A^ j^ M^rtlW^^ fM^^^AIff^^^A
Al»
Mr
W«ttr
Land
Uvfng RMOUTCM
|| RESPC
//TfafTRfltJQfl
_ — - -- - - • ^^~-
Scc*t»l AMpeniM
(OaeUcnf-AeOafm)
)NSES
f
Economic tnd
EnvtroniiMntai
Agents
AOTwiMuunra
Lirt.^ahnlr^
nwMwnoraB
EntVfpnSAS
lr««madonaJ
1
Socwta/ Hmfonmt {Dfdmom - Action*)
Figure 3-1. OECD Pressure-State-Response Framework (Adapted from "OECD Core
Set of Indicators for Environmental Performance Review,"
Environmental Monograph No. 83 (1993).
'"'Causal" frameworks seek to organize or classify environmental information in terms of the aggregate causal
flow or "cycle" or human-environment interactions (USEPA, 1995).
Revised Draft
May 1996
-------
Frameworks for Developing Indicators
Within the PSR framework, three broad types of indicators, and potential subcategones.
can be distinguished;
• Indicators of Environmental Pressure describe the pressure that human activities
exert on the environment, including the quality and quantity of natural resources. The
subcategones of this indicator type are indicators of direct pressures (pressures exerted
directly on the environment that are normally expressed in terms of emissions or
consumption of natural resources), indirect pressures (background indicators reflecting
human activities that lead to direct environmental pressures), and underlying societal
pressures (social and technological forces that drive human activities).
• Indicators of the State of the Environment relate to the quality of the environment
and the quality and quantity of natural resources. As such, they reflect the ultimate
objective of environmental policy making. Indicators of environmental conditions
should measure the state of the environment .and changes in that state over time, rather
than the pressures on the environment. In practice, however, the direct measurement
of environmental conditions can be difficult or very costly. Therefore, the
measurement of environmental pressures is often used as a substitute for the
measurement of environmental conditions. Indicators of the state of the environment
can be subcategorized by nested spatial scales (local, regional, and global ecosystems;
human health and environment-related welfare) and by biological, chemical, physical,
and ecological functions and variables.7
• Indicators of Societal Responses relate to individual and collective actions to mitigate.
adapt to, or prevent human-induced damage to the environment and to halt or reverse
environmental damage that has already occurred. Societal responses also include
actions for the preservation and the conservation of the environment and natural
resources. Indicators of societal response can be subdivided by the type of responding
entity (e.g., governments, private sector, individuals, or partnerships).8
Each type of indicator has advantages, provided that it is appropriate for the target audience and
effectively meets the goals and objectives of the project, and disadvantages.
Pressure indicators are particularly useful in formulating short-term (i.e., annual)
objectives and in evaluating short-term (i.e., annual) performance, because they explore potential
cause and effect relationships between human activities and the environment (e.g., whether
increasing or decreasing emissions are associated with changes in ambient conditions). They can
'USEPA, op. cit.
"Ibid. 5. 8, 9, and 10.
Revised Draft
May 1996
-------
Process for Selecting Indicators
m many coses be relatively easy to assemble, maintain and update, and are easy to interpret. A
fairly extensive data base is available to construct pressure indicators because of the widespread
environmental monitoring and regulatory compliance framev/ork established in the United Scutes
(e.g.. many emissions are regularly monitored). Because pressure indicators are often developed
from direct measurements or model-based estimates, they can provide direct feedback on whether
policies are meeting project goals (e.g., reduce total nitrogen discharges by 40% from all direct
dischargers)."0'" One disadvantage of pressure indicators is that it is sometimes difficult to
establish a causal link between the pressures exerted and die state of the environment without
additional information.
State of the environment indicators are crucial for a long-term evaluation of the
environment and environmental programs.1213'14 Preparing these indicators, however, can be
difficult. Data on ambient environmental conditions (e.g., amount of old growth forest,
concentration of mercury in water) are often limited in temporal or geographic scope, difficult
to locate, or confusing to interpret. Techniques to measure actual environmental conditions (i.e.,
state) can be difficult and costly and must occur over an extended period (i.e., there is often a
lag time after a control action is taken before measurable chiinges to the state of the environment
occur). Nevertheless, continued efforts to develop such indicators are being made and are
needed. Without them, no firm conclusion can be reached about the effectiveness of current
policies in protecting and improving the state of the environment.
Societal response indicators are useful because they provide a measure of the scope of and
level of participation in environmental protection programs (e.g., number of dischargers affected
by and level of compliance with government regulations). Response indicators are limited,
"Hammond, op. cit
'"State Environmental Goals and Indicators Project, op. cit.
"USEPA, op. cit.
i:Hammond, op. cit.
"Ibid.
Revised Draft
May 1996
-------
Frameworks for Developing Indicators
however, because they do nor directly measure what is happening to the environment. Ideally.
response indicators should be developed after and be closely integrated with pressure and state
indicators to provide a complete picture of the issue being studied.15'16'17
The PSR framework can .be modified to suit a particular environmental indicator
development project. For example, an expansion of the PSR framework has been suggested for
developing a system of environmental statistics and indicators. The new version includes
"Effects" as a category to describe relationships between two or more pressure, state, and/or
response variables. Appendix A contains additional information on the PSR/E framework.
Because frameworks provide the context for organizing indicators and associated data, flexibility
is necessary to ensure that the framework adopted best meets the goals and objectives of the
specific project or use for which it is being employed.
1JIbid.
"Ibid.
I7USEPA. op. cit.
Revised Draft 9 May l996
-------
4.0 PROCESS FOR SELECTING ENVIRONMENTAL INDICATORS
Presented in this section is a step-wise approach for selecting indicators, based on standard
decision-making practices. Three primary steps are described; they include:
• Identifying and recommending indicators
• Identifying and recommending data to support indicators
• Selecting final indicators. .
Examples appear throughout this section to demonstrate the application of the selection process.
A description is provided for developing an Indicator Tearn to best perform the recommended
process.
Different approaches can be used to select environmental indicators. One approach is to
select indicators for a particular application on an ad hoc basis from existing indicators and/or
readily availabfe data. Although this approach can yield meaningful and informative indicators,
indicators developed from existing indicators and/or available data can have limitations:
• The range of possible indicators is limited to indicators or data developed previously
for other purposes.
• Available indicators or data may not be directly related to or appropriate for the
intended message, goals, or objectives of the new indicator project. Yet, there may be
a bias to try to "force fit" available information.
• Available indicators or data may not be suitable lor the target audience.
i
• Using previously developed indicators or data may result in confusing or mixed
messages, as well as indicators that are neither relevant, nor representative.
Another approach uses a systematic process in which indicator selection is based on
postulated cause-effect linkages between valued environmental attributes and the societal and
Revised Draft
10
May 1996
-------
Process for Selecting Environmental Indicators
natural factors chat potentially affect these attributes."1 This approach suggests that it is
important to define, at the beginning of the effort, the overall goals and objectives of the
indicator'project, the intended message for the indicator, the framework (i.e., pressure, state.
response) for presenting indicators, and the target audience before selecting indicators and
evaluating data availability. Although this approach may expose gaps in existing data, it allows
society's environmental values and current scientific understanding of environmental linkages to
drive indicator selection. The identification of such data gaps can drive further research and data
collection. The remainder of this section provides more detail on the latter approach.
N
PROCESS DESCRIPTION
The indicator and data selection process presented in this document is meant to be flexible
and should be modified to best meet the needs -of the particular indicator development activity.
The tikeiihood of a successful outcome will be increased, however, if the following fundamental
principles of decision making are followed:19'20
• Initiate the process by clearly identifying goals and objectives (including defining the
theme and target audience for the indicator)
• Identify candidate indicators that support the identified goals and objectives
• Develop and apply a decision making process for selecting the best indicators.
In addition, developing successful environmental indicators requires leadership and a
feedback mechanism. The project will need a lead group (i.e., Indicator Team) that is responsible
for initiating and coordinating the project, as well as providing leadership throughout the process.
The Indicator Team should involve a representative and balanced stakeholder group of interested
and affected parties throughout the process. Stakeholders can be involved in many ways.
"For discussions of this type of approach, see, for example. Managing Troubled Waters: The Role of Regional
Marine Environmental Monitoring (National Academy Press. 1990) and the Indicator Development Strategy for the
Environmental Monitoring and Assessment Program (USEPA, 1994).
'thang, R.Y. and P.K. Kelly. 1993. Step-By-Step Problem Solving. Irvine, CA: Richard Chang Associates,
Inc.
:ilChechile. R.A. and S. Carlisle. 1991. Environmental Decision Making: A Multidisciplinary Perspective. New
York: Van Nostrand Reinhold.
Revised Draft 11 May 1996
-------
Process for Selecting Indicators
including participation in team meetings, workshops, or othe- facilitated sessions; and inclusion
in product review and comment cycles. Stakeholders will vary by indicator project, but may
include the following types of individuals:
• Subject matter experts
• Information and data providers
• Outside professionals (e.g., consultants and representatives from industry, public interest
groups, nongovernmental organizations, academe, and individuals with financial and
economic backgrounds)
• Decision makers (e.g., government policy makers)
• Customers (e.g., educators, media representatives, general public).
s
Involving stakeholders in the indicator development process is crucial for achieving buy-
in, ensuring that the indicator is on target with the desired goals and objectives and message or
theme and that the indicator is understandable and effectively promoted. Indicators developed
in the absence of stakeholders may be stalled, derailed, or refuted.
The remainder of this section discusses the three steps in selecting environmental
indicators: identifying indicators, identifying data for supporting indicators, and selecting final
indicators. Figure 4-1 is a flow chart illustrating the indicator selection process.
STEP 1: IDENTIFY AND RECOMMEND INDICATORS
This step results in a list of candidate indicators that serve the goals and objectives of
the indicator project, support the framework for communicating the theme or message
of the project, and meet the evaluation criteria developed to aid in indicator selection.
The first task of the Indicator Team is to define the goals and objectives of the
indicator project and develop a theme and framework that will be used to
communicate to the target audience. Next, the Indicator Team develops a draft list of
potential indicators. Then, the team selects candidate indicators using specific
evaluation criteria.
Revised Draft 12 May 1996
-------
Process for Selecting Environmental Indicators
Indicator Team
and
Stakeholders
Stepl: Identify and
Recommend Indicators
Task la: Define goals and
objectives of indicator
Task Ib: Identify possible
indicators that support
indicator goals and
objectives and organize
them according to the
message elements.
Task Ic: Evaluate each
possible indicator to
identify candidate
indicators
Step 2: Identify and
Recommend Data to
Support Indicators
Task 2* Identify
potential data sets for
each indicator
Task2b: Evaluate
candidate data sets
Task2c: Address data
Step 3: Select Final
Indicators
Figure 4-1. Process for Selecting Indicators
Revised Draft
13
May 1996
-------
Process for Selecting Indicators
Task la: Define goals, objectives, and framework of the indicator
The following questions guide the development of indicator goals, objectives, and
theme or message:
• Why is the indicator being developed?
• What is its intended use (e.g., track progress toward meeting program goals,
develop an objective description of the state' of the environment, educate the
general public)?
• Who will use the indicator (e.g., program manager, scientists, educators)?
• Who is the intended audience for the indicator (e.g., program managers,
legislators, the general public)?
' • What are society's goals, values, and concerns? Are they addressed by the
indicator?
* Is the indicator related to a program mission or goal statement(s)?
• Does the indicator objectively communicate information to the target audience?
The answers to these questions provide the essential foundation for effective indicator
development. These questions should be considered at project initiation and referred to
throughout the entire process. They are critical at the 'outset to establish the scope and
approach of the indicator project. They are also central to the development of individual
indicators. An effective indicator, whether used individually or in a group, must have a
clear message relevant to the target audience and intended use of the indicator. By
carefully answering the goal and objective questions, the Indicator Team will be better
able to implement an efficient and effective process. Stakeholders can contribute during
these early stages to help define goals and objectives, especially when identifying key
messages and audiences.
Although defining the goals and objectives is critical to effective, targeted indicator
development, defining a framework is central to the indicator selection process. The
framework serves as the organizational structure for the indicator project. The most
Revised Draft 14 Ma? l996
-------
Process for Selecting Environmental Indicators
common framework used to date has been the Pressure-State-Response framework.
described briefly in Section 3.0.
An indicator topic may be narrow and require few indicators or be broad and need many
indicators to tell the entire story. If the topic is broad or complex and requires many
indicators, it may be necessary to develop an outline or a list of all information needed
to convey each element of the framework. This step is similar to preparing an outline for
a story. Each individual indicator should relate closely to the overall goals and
objectives. Exhibit 4-1 provides an example of developing the goal, objective, and
elements of the PSR framework for an air pollution topic.
Exhibit 4-1. Example Application of Task I (a)
Define the goals, objectives, and framework of the indicator project
Goal: Raise public awareness about air pollution.
Objective: Communicate to the public the role of individuals in air pollution.
Framework:
1. Pressure • Mobile sources affect air quality
• Mobile source emissions are significant compared to other sources
2. State • Ambient concentrations of pollutants associated with mobile sources
3. Response • Behavioral changes
• Technological changes
Task Ib: Identify possible indicators that support indicator goals and objectives.
After identifying the overall goal and objectives, the next step is identifying potential
indicators. The potential indicators are developed based on their effectiveness in
conveying the elements of the framework, while ensuring that they are suitable for the
intended use of the indicator and target audience. Through one or more meetings,
workshops, or other events, supplemented with review and comment cycles of written
materials, the Indicator Team and stakeholders should develop a comprehensive list of
specific indicators. Next, the Indicator Team may want to examine currently used
indicators for additional ideas. The potential indicators can then be grouped according
to the elements of the framework, as illustrated in Exhibit 4-2.
Revised Draft 15 May 1996
-------
Process for Selecting Indicators
Exhibit 4-2. Example Application of Task l(b)
Identify possible indicators and organize them according to the framework.
• Pressure—Mobile sources affect air quality
» N'umber and type of vehicles on roads
> Number of vehicle, miles driven per year
*• Emissions of selected pollutants from mobile sources
* Pressure—Mobile sources are a significant scarce of air polli.tion compared to other sources
» Emissions of selected parameters from mobile sources compared to stationary sources •
• State—Concentration of vehicle emissions in the environmem;
» Atmospheric concentration of selected pollutants
* Concentration of selected pollutants in soil near roads and highways
» Concentration of selected pollutants in water
• Response—Extent of• behavioral and technological changes
t
» Number of individuals participating in car pools
- Extent of mass transit opportunities
fc Emissions comparison between regular and fuel-efficient vehicles
» Availability of fuel-efficient vehicles
*
-.1
"I
Task Ic: Evaluate each potential indicator to identify candidate indicators.
The comprehensive list of potential indicators developed in Task Ib should be narrowed
to the indicators best suited for the project. The Indicator Team, with the stakeholder
group, should now develop the evaluation criteria appropriate to the specific indicator
development project so that the criteria can be applied to select candidate indicators from
the list of potential indicators. The process of selecting candidate indicators should be
well-documented so that it is understandable to participants and outside reviewers and can
be reproduced. Section 5.0 describes an approach for identifying appropriate selection
criteria.
Before applying .the evaluation criteria, the Indicator Team should ensure that each
member has the same understanding of each criterion. The Indicator Team may want to
use specific questions or examples for each criterion to facilitate evaluation of the
candidate indicators. For example, if the criterion is "understandability," the following
questions might be helpful:
• Is the information of the right technical level for the target audience?
Revised Draft
16
May 1996
-------
Process for Selecting Environmental Indicators
• Is the proposed display and presentation effective and appealing?
• Is the methodology used to create the indicator well-documented and understandable
so that it can be easily communicated and reproduced?
Table 5-1 in Section 5 provides additional examples of evaluation criteria.
The Indicator Team should also decide on the approach to be used for applying the
evaluation criteria. The evaluation criterion, for example, can be weighted equally so that
possible indicators are simply ranked (quantitatively or qualitatively) according to how
well they meet each criterion. Alternatively, each criterion can be weighted to emphasize
its relative importance compared to the others. In addition, the evaluation criteria can be
grouped into essential criteria (i.e., criteria an indicator must meet) and preferable criteria
(i.e., criteria an indicator should meet if possible).21 This idea is discussed in more
detail in Section 5.
The result of Task Ic is a list of candidate indicators: the best indicators among those
identified for conveying the message.
STEP 2. IDENTIFY AND RECOMMEND DATA TO SUPPORT INDICATORS
Once the candidate indicators are selected, the next step is to identify the data that will
be used to quantify the indicators. This requires examining existing data collection and
analysis programs to determine whether appropriate data are or will be available. The
process can be accomplished using the following two steps.
Task 2a: Identify potential data sets for each indicator.
The Indicator Team should conduct a focused search to identify candidate data sets that
will support the candidate indicators. The Indicator Team may work with the stakeholder
group and other subject matter and/or data experts to identify potential data sets held by
EPA, other Federal agencies, and other entities. A literature review may also be helpful.
21 State Environmental Goals and Indicators Project, op. cit.
Revised Draft
17
May 1996
-------
Process for Selecting Indicators
Data sets should be identified and grouped by the candidate indicator they could be used
to generate.
Task 2b: Evaluate candidate data sets.
After potential data sets are" identified, they can be evaluated to select the most suitable
data sets. The Indicator Team and stakeholder group may want to score data sets using
pre-determined evaluation criteria. Section 6.0 discusses criteria for selecting appropriate
data sets. The Indicator Team may choose one or more of a variety of ways to apply the
data criteria, including weighted scoring, completion of a checklist based on the criteria,
and best professional judgment. The approach should be well-documented so that it is
understandable to participants and outside reviewers and can be reproduced. It may be
desirable to complete summary forms for candidate data sets to facilitate decision making.
Appendix B presents an example summary form.
Several data sets might be appropriate for use in generating a particular indicator. If any
of several data sets could be used, all would be considered unless one or more were
clearly inferior to the others (i.e., being similar on most criteria but clearly worse on
some). It may be necessary, therefore, to develop preliminary screening criteria, in
addition to the detailed data evaluation criteria, to quickly eliminate the less suitable
candidate data sets prior to a thorough evaluation. Such preliminary screening may be
necessary if resources are limited.
After applying the evaluation criteria, if several dafci sets are found to be appropriate for
use in generating a particular indicator, the best one would generally be chosen. In some
cases, it might not be possible to identify any appropriate data. A data gap exists if no
data are available or if the available data are inadequate and cannot be improved.
Task 2c: Address data gap.
For indicators that lack adequate data, the Indicator Team may take the following actions:
(1) Document the data gaps.
Revised Draft
18
May 1996
-------
Process for Selecting Environmental Indicators
(2) Review other existing indicators and data to determine whether one or more can be
used as interim substitute indicators to at least provide some information on the
factors to be addressed by the candidate indicators for which no adequate data are
available.
(3) Develop strategies for filling the identified data gaps, including improvements to
existing programs of data collection, data analyses, and information management.
Developing strategies includes determining whether data can be made available by
modifying existing data management and analysis procedures. For example, this
could include the reanalysis of existing data or the integration of two or more separate
data sets.
a. If the needed data can be made available by changes in existing data management
or data analysis procedures, develop a strategy for making the needed changes.
b. If the needed data cannot be made available by changes to existing data
management or data analysis procedures, determine whether there are validated
test methods, statistical methods, etc. at the levels of accuracy and levels of
reliability required:
- For each indicator for which validated methods are available, identify what data
is required and design a data collection program (of appropriate statistical
design) and a data analysis program. If feasible, implement the program.
- For each indicator for which validated methods are not available, set up a
process to develop such methods. If needed, set priorities for developing
these methods. Once appropriate methods are developed, identify what data
are required and design a data collection program (of appropriate statistical
design) and a data analysis program. If feasible, implement the program.
STEP 3: SELECT FINAL INDICATORS
After identifying the candidate indicators and the data sets available to support each
indicator, the Indicator Team will need to select the final indicators. At this stage, the
indicator team has full knowledge of which indicators best serve the goals and objectives
of the project. The team has also evaluated the quality of available data and identified
data gaps. Now the Indicator Team must work with this information to select final
indicators for the project.
The procedure for selecting final indicators will likely be an iterative process using
stakeholder involvement, peer review, and expert knowledge. The approach for selecting
Revised Draft 19 MaX l996
-------
Process for Selecting Indicators
final indicators used by the State Environmental Goals and Indicators Project (SEGIP)
involved the application of decision criteria called qualification standards. These are
listed in Exhibit 4-3. After applying the standards and identifying final indicators, the
participants realized that the qualification standards did not yield a sufficient number of
indicators to meet the needs of the project. In response, they developed a three-tiered
classification. The first tier, Type A indicators, meet the qualification standards. The
next tiers, Type B and Type C, classify the remaining indicators according to the
availability of data or the level of 'effort required to develop the data needed to support
the indicators. The definitions of Type A, B, and C indicators are provided in Exhibit
4-4.
The process of making a preliminary choice, gatheriag more information, and making a
more refined choice of indicators is iterative. Additional iterations may be necessary to
refine the selection of indicators and to incorporate new information as it is gathered.
Exhibit 4-3. State Environmental Goals and Indicators Project Qualification Standards22
t. The indicator was national in scope and could be consistently oisplayed at the state level.
2. The indicator met SEGIP Essential Indicator Selection Criteria.
3. The indicator currently existed and was available to the states.
4. The indicator reflected a direct environmental value and not an administrative or program result.
Administrative measures that summarized counts of definable environmental degradation (e.g.,
exceedances. spills) were acceptable.
5. The indicator supported an environmental result relevant to the U.S. EPA-State relationship as
envisioned in the proposed Performance Agreements.
::State Environmental Goals and Indicators Project, op. cit.
Revised Draft 20 Mav l996
-------
Process for Selecting Environmental Indicators
Exhibit 4-4. State Environmental Goals and Indicators Project Classification Scheme"
Type A: Indicators for which adequate data are available now and can be used to support the
indicator without significant additional cost considerations. To be classified as Type A. an indicator:
* Meets all essential selection criteria and most preferred criteria.
• Is presently available for use in its present condition, and
• Can be acquired easily at little or no cost.
Type B: Indicators which are presently feasible, but cannot be provided due to inordinate cost,
analytical complexity, or time constraints. Type B indicators are those that, could be made available
now if some operational barrier can be overcome. The data needed to produce the indicator exist but
because of cost concerns, analytical difficulties, time constraints, manpower issues, or some other
impediment, the indicator cannot be provided.
Type C: Prospective indicators for which there is no reasonable prospect of development without
some extraordinary expenditure of resources. Type C indicators are purely prospective. The data do
not exist and there is no clear intent to collect them. Type C indicators exist as designs only.
"State Environmental Goals and Indicators Project, op. cit.
Revised Draft 21 Ma? l996
-------
5.0 CRITERIA FOR SELECTING ENVIRONMENTAL INDICATORS
This section presents a list of evaluation criteria that was adapted from a review of other
environmental indicator projects (e.g., Intergovernmentiil Task Force on Water Quality
Monitoring, State Environmental Goals and Indicator Project, Environmental Monitoring and
Assessment Program, and the International Joint Commission for the Great Lakes) and refined
to provide a comprehensive list of criteria that could be used for a variety of indicator selection
projects. Table 5-1 provides these criteria and defines each criterion with one or more specific
questions. It is often useful to define the criteria as a series of questions, because questions can
be easier to apply when selecting indicators than narrative definitions. Appendix C presents
examples of selection criteria used by various organization.;.
Choosing clear evaluation criteria and determining an effective means of applying them
are two critical aspects of the overall process of selecting environmental indicators. The
evaluation of possible indicators against criteria enables the: Indicator Team and stakeholders to
narrow a potentially broad list of indicators to those that are optimally suited for achieving the
project goals and objectives. Applying well-defined evaluation criteria will help focus the
selection process and reduce potential bias by providing a clearly articulated and relatively
objective means to evaluate, or score, possible indicators. This process should ideally be
cooperatively developed by the Indicator Team with support from a representative and balanced
stakeholder group. The value of developing criteria that aie clearly defined and understandable
cannot be overstated. Everyone applying them will then be employing the same working
definition.
Evaluation criteria should be determined relatively early in the process, not long after
defining possible indicators. The Indicator Team and stakeholder group should identify
evaluation criteria that will effectively reflect the goals and objectives of the indicator project.
Brainstorming techniques can be used to develop a broad hst of potential evaluation criteria that
can be refined to present a workable list of final criteria. This section describes the process of
determining and applying indicator evaluation criteria.
Revised Draft 22 Ma? 1996
-------
Criteria for Selecting Environmental Indicators
Table 5-1. Suggested Evaluation Criteria for Selecting Indicators
Criterion
Definition
Validity
Social and Environmental
Relevance
Appropriate Scale
Integration of Multiple Impacts
Representative
Sensitivity
Interpretability
[nterpretable
Trend Evaluation
Timeliness •
Timely/ Anticipatory
Understandability
Understandable
Documented
Consistency
Provision of Decision Support
Does the indicator express society's environmental values, goals, and
concerns by presenting information relevant to a desired policy goal, issue.
legal mandate, or agency mission? Does the indicator reflect the project
message? Can this information be understood by and easily related to the
general public and decision makers? Is the indicator seen by the target
audience as being important or relevant to their lives?
Does the indicator respond to changes on an appropriate geographic (e.g.,
global, national, regional, or local) and temporal (e.g., daily, monthly,
yearly) scale?
Does the indicator represent the cumulative impacts of multiple stressors
(e.g., water quality affected by nonpoint source discharges, point source
discharges, acid rain, erosion)? Is it broadly applicable to many stressors
and sites?
Are changes in the indicator highly correlated with changing trends in the
information it is selected to represent (e.g., is an indicator of industrial
loadings to surface water highly correlated to declining surface water
quality)? Does the indicator present an accurate picture for the message it
is intended to convey?
Can the indicator distinguish small changes in environmental conditions
with an acceptable degree of resolution (e.g., will the indicator respond to
modest changes such as occasional permit violations, new plants coming
online, or gradual improvements in quality over time)?
Is there a reference condition or benchmark for the indicator against which
to measure changes and trends (e.g., standards, limitations, criteria, goals)?
Has the data for the indicator been collected over a sufficient period of
time to allow analysis of trends or provide a baseline for estimating future
trends?
Does the indicator provide early warning of changes?
-
Is the indicator appropriate for the target audience? Is the indicator
presented in a format tailored to the needs of the target audience? Is it
simple and direct?
Is the methodology used to create the indicator well-documented and
understandable so that it can be easily communicated and reproduced?
Is the information presented by the indicator consistent over time (e.g., are
definitions, measurement techniques, and analytical methodologies
consistent and comparable)?
Is the level of information by the indicator appropriate for the target
audience to use in decision making?
Revised Draft
23
May 1996
-------
Process for Selecting Indicators
Table 5-1. Suggested Evaluation Criteria for Selecting Indicators (continued)
Criterion
Definition
Cost Considerations
Cost Effectiveness
Minimal Environmental Impact
Measurable
Data Availability
Is data to support the indicator readily available? Can it be obtained with
reasonable cost and effort? Cai it be reproduced, maintained, or updated?
Do sampling procedures produce minimal environmental impact?
Does the indicator measure a feature of the environment that can be
quantified simply, using standard methodologies with a known degree of
accuracy and precision?
Are adequate data available for immediate indicator use? Do constraints
exist on data collection that require postponement of indicator
development?
DETERMINING APPROPRIATE EVALUATION CRITERIA
The process of identifying evaluation criteria is flexible and will vary to best meet the
needs of the particular indicator project. The choice of criteria is often driven by the intended
use for the indicators. For example, indicators that will be used to evaluate the effectiveness of
management measures for improving fish species diversity and population size may require
criteria that emphasize assessing scientific validity and accuracy. Indicators intended for
communication to the general public may need criteria that stress assessing presentation
»
effectiveness (e.g., clarity and simplicity). In all situations, however, several key features are
critical to developing effective criteria, including the following:
* Criteria reflect project goals and objectives
• Criteria are clearly defined and understandable to all involved parties (it is very
important that all participants apply the same definition)
• Criteria are sufficiently well-defined to avoid ambiguity (e.g., sometimes it is helpful
to provide examples, and/or parameters, ranges, or other measures to define criteria and
the extent to which they are met)
• Criteria are practical, valid, and legitimate.
• Criteria are nonbiased.
The evaluation criteria presented in Table 5-1 are grouped according to validity,
interpretability, timeliness, understandability, and cost considerations:
Revised Draft
24
May 1996
-------
Criteria for Selecting Environmental Indicators
Validity—Indicators should be valid measures of the valued attribute. Validity is
defined here as a close qualitative or quantitative link between the attribute actually of
interest (e.g., biological integrity) and the measurable quantity represented by the
indicator. Several factors listed in Table 5-1 contribute to a close logical link between
the indicator and the attributes of societal concern or value:
*• Indicators that respond at the appropriate spatial and temporal scales are more
likely to be valid measures of an attribute of concern.
*• If the purpose of an indicator is to assess environmental status and trends,
indicators that respond to cumulative effects of multiple stressors will be more
representative of the overall ecosystem condition than those that are responsive to
only a few stressors.
•> Indicators that are highly correlated with other measures (of a specified attribute)
will'tend to be representative of the environmental attribute or system being
measured.
» Indicators must be sensitive enough to measure changes over a reasonable time
but not so sensitive that they fluctuate substantially between time periods. The
signai-to-noise ratio for an indicator is determined in part by the data used to
generate the indicator. Expert knowledge and peer review can be used to assess
the sensitivity of different indicators.
Interpretability—Indicators should be interpretabte in terms of the end point in the
.assessment process. They should be able to distinguish unacceptable from acceptable
environmental conditions. Ideally, each indicator will have a benchmark against which
to measure change.
Timeliness—Timely indicators that anticipate future changes in the environment are
preferred over those that are not anticipatory. To the extent that an indicator does not
anticipate future conditions, the indicator with the least time lag would be preferred.
The time lag depends on both characteristics of the indicator and the time lag between
the data collection and when the data are available to calculate the indicator.
Understandability—Indicators should be geared toward the target audience. Since so
many indicators are used for public outreach, indicators should be understandable by
the public and perceived as relevant Understandability is in part a characteristic of the
indicator and in part a function of how the indicator is presented. EPA may need to
educate the public on the importance of some indicators. If possible, indicators should
be "attention grabbers" in that they reflect the values of the audience (e.g., information
on the number of fish in a water body is generally more interesting to the public than
data on macroinvertebrates lower in the food chain). Keeping data presentations
simple, graphic, and consistent enhances indicator Understandability. The use of focus
groups may help EPA to understand how the public perceives the indicators and may
provide insights on ways to improve the indicator. Involving a representative and
Revised Draft 25 Ma? l996
-------
Process for Selecting Indicators
balanced stakeholder group throughout the indicator selection process should improve
indicator, presentation and understandability.
• Cost Considerations—Indicators should be cost effective relative to alternatives and
to the effort and expertise, required to collect the'data, if required, and report the
indicator over time.
The key to effective evaluation criteria is that they are appropriate for the particular
indicator project. Whenever possible, criteria should be tefined to best meet the needs of the
specific project. Evaluation criteria that are targeted to the project goals and objectives, intended
use, and target audience, are easier to apply than more generic criteria. For example. Table 5-2
demonstrates one way that the "validity and interpretability" criteria presented in Table 5-1 might
be refined to better meet the needs of a particular indicator project related to surface water
quality.
APPLYING THE CRITERIA
Evaluation criteria can be applied using a variety of techniques to rank the possible
indicators. The Indicator Team should select or designate an approach best-suited for that
particular indicator project. Whatever approach is selected, it should be documented so that it
is understandable to outside parties and can be reproduced easily.
One of the most common techniques is to weight the relative importance of various
evaluation criteria and to score possible indicators according to how well they fulfill the weighted
evaluation criteria. This approach, referred to as a weighted numeric index, is based on the
following steps:
• Identifying arid assigning weights to criteria
• Applying the criteria to the indicators
• Choosing the indicators) with the highest weighted score.
Revised Draft
26
May 1996
-------
Criteria for Selecting Environmental Indicators
Table 5-2. Example Development and Application of Suggested Evaluation Criteria
This case study demonstrates the development of project-specific evaluation criteria based on the target audience.
purpose.-and indicator goals. This example is created using the "validity and inierpretabtlity" evaluation criteria
described in Table 5-1.
Background
Project: Environmental Indicator Bulletins-Surface Water Quality
Audience: General Public and Decision Makers
Purpose: Provide audience with a yearly assessment of national surface water quality. '•
Format: Presented in a four-page color bulletin with limited space for graphics.
Procedure
Following the process outlined in Chapter 4.0 of this document, the Indicator Team, with the close support of
the stakeholder group, developed goals and objectives for the indicator project They also identified possible
indicators using brainstorming techniques. In addition, brainstorming was used to determine selection criteria,
were then refined and tailored, using consensus-building techniques, to best meet the needs of the project.
Assess human and natural impacts, current conditions, and actions to improve water quality at a national
level.
Goal:
Criteria
Validity
Social and Environmental
Relevance •
Appropriate Scale
Definition
Example of Project-Specific
Criteria
Integration of Multiple Impacts
Does the indicator express society's
environmental values, goals, and
concerns by presenting information
relevant to a desired policy goal,
issue, legal mandate, or agency
mission? Is the indicator seen by
the target audience as being
important or relevant to their lives?
Does the indicator respond to
changes on an appropriate
geographic (e.g., national or
regional) and .temporal (e.g.. yearly
or biennially) scale?
Does the indicator represent the
cumulative impacts of multiple
stressors (e.g.. water quality
affected by nonpoint source
discharges, point source discharges,
acid rain, erosion)? Is it broadly
applicable to many stressors and
sites?
Does the indicator reflect the goals
of the Clean Water Act?
Does the indicator provide national
representation of surface waters?
Are data supporting the indicator
appropriate to report on a national
scale?
Does the indicator integrate
impacts from agricultural runoff,
silviculture, construction activities.
point source discharges, and runoff
from nonpoint sources?
Does the indicator capture all types
of land uses?
Revised Draft
27
May 1996
-------
Process for Selecting Indicators
Table 5-2. Example Development and Application of Suggested Evaluation Criteria
(continued)
Criteria
Definition
Example of Project-Specific
Criteria
Validity
Representative
Are changes in the indicator highly
correlated to changing trends in ihe
information it is selected to
represent (e.g.. is an indicator of
industrial loadings to surface water
highly correlated to declining
surface water quality)? Does the
indicator present an accurate model
of the message it is intended to
convey?
Will the indicator respond to
changes in other factors affecting
water quality?
Does the indicator accurately
reflect national surface waters?
Does the indicator separate surface
water from ground water? Is the
information presented in the
indicator indicative of surface
water only?
Sensitivity
Can the indicator distinguish smiill
changes in environmental conditions
with an acceptable'degree of
resolution (e.g., will the indicator
respond to modest changes such as
occasional permit violations, or new
plants coming online)?
Will the indicator respond to
modest changes to environmental
conditions (e.g., a 10% increase in
national nitrogen loadings from
atmospheric deposition)?
Interpretability
Interpretable
Is there a reference condition or
benchmark for the indicator against
which to measure changes and
trends (e.g., standards, limitations,
criteria, goals)?
Are there standards or other
benchmarks for the indicator?
Are there water quality criteria for
all of the parameters reported in
the indicator?
Trend Evaluation
Has the data for the indicator been
collected over a sufficient period of
time to allow analysis of trends ar
provide a baseline for future trends?
Has ambient surface water quality
monitoring data been collected for
over 10 years?
Are there sufficient, accessible,
reliable historical surface water
monitoring data that can be used to
establish a baseline?
Identifying and Assigning Weights to Evaluation Criteria
The Indicator Team and stakeholders should identify criteria and rank them in terms of
their importance in relation to each other. After ranking, the Indicator Team should apply a
numeric weight to each of the ranked criterion. This can be done by assigning a percentage to
each criterion so that all the criteria together total 100 percent. Although criteria sometimes may
be weighted equally, it is often more effective to assign different weights so that the criteria
Revised Draft
28
May 1996
-------
Criteria for Selecting Environmental Indicators
accurately reflect the needs of the particular indicator project. If the Indicator Team has
difficulty assigning numeric weights, it can group the criteria into essential (i.e., criteria an
indicator must meet) and preferable (i.e., criteria an indicator should meet if possible)
categories/4 or use some other type of qualitative ranking approach. Brainstorming and
consensus-building techniques should be used throughout this process. Table 5-3 provides an
example of this approach.
Table 5-3. Example Approach for Using Weighted Criteria
for Evaluating Possible Indicators
Criteria
Validity
InterpretabUity
Timeliness
Understandability
Cost Effectiveness
Total
Weight
40%
20%
10%
20%
10%
100%
Ratine Scale: 1 to 10
Possible Indicators
Indicator 1
2 (0.8)
3 (0.6)
8 (0.8)
7 (1.4)
5 (0.5)
4.1
Indicator 2
4 (1.6)
6 (1.2)
1 (0.7)
. 3 (0.6)
1 (0.1)
4.2
Indicator 3
8 (3.2)
6 (1.2)
4 (0.4)
6 (1.2)
7 (0.7)
6.7
Applying the Criteria
The Indicator Team should rate each possible indicator against each' criterion on a scale
(e.g., 1 to 10, with 10 being the highest). Then the scores can be determined using the weighting
factors. A typical numeric index approach uses such formulas as the following:
• Additive Model
Score = (S, x Wt) + (S2 x W2) + . . . (Sn x Wn)
Where:
S = Score assigned to each indicator for a particular evaluation criterion
W s Weight assigned to the criterion.
:4State Environmental Goals and Indicators Project, op. cit.
Revised Draft
29
May 1996
-------
Process for Selecting Indicators
• Multiplicative Model
• Score = (S, x W,) x (S, x W,) x . . . (Sn x Wn)
Where:
S = Score assigned to each indicator for a particular evaluation criterion
W = Weight assigned to each criterion.
Additive models tend to equalize the influence of all factors, whereas multiplicative models tend
to emphasize the differences among factors. As a result, an additive model tends to produce
scores within a narrow range; a corresponding multiplicative approach generates a much wider
range of scores. Weighting the scores using either the additive or multiplication model produces
a numeric index for each possible indicator. Additional information on applying numeric indices
is presented in Chechile and Carlisle (1991) and Chang and Kelly (1993). The following list
briefly summarizes selected advantages and disadvantages of this approach:
• Advantages
»• A numeric index can be based on quantifiable criteria important to the indicator
selection process.
» The index can be developed with input from different sources and easily modified
so that the information can be tailored to serve a variety of indicator projects.
+ The approach is straight-forward, with results that are standardized and
reproducible.
• Disadvantages
> The more complex the index, the more difficult it is to apply, reproduce, and
explain to the public.
*• Care must be taken in constructing the index to ensure that the correct criteria are
chosen and weighted appropriately; the wrong choice of criteria and/or weighting
factors may result in a poor index.
» The range of scores may end up too small to allow for choosing between
indicators.
Revised Draft
30
May 1996
-------
Criteria for Selecting Environmental Indicators
To minimize the potential disadvantages of this approach, the Indicator Team may want to
compare weighted results to unweighted scores. Also, if any of the resulting rankings seem
inappropriate (e.g., an indicator that was believed to be good does not make the list or
questionable indicators are ranked high), the Indicator Team may need to reexamine the
evaluation criteria. It is always helpful to fully test the criteria before using them to select
indicators.
Re vised Draft 31 May 1996
-------
6.0 CRITERIA FOR SELECTING EXISTING DATA SETS TO
SUPPORT INDICATORS
The basis for all environmental indicators is data. Sections 4.0 and 5.0 presented an
approach for selecting environmental indicators based on defining, at the outset, the overall goals
and objectives of the indicator project, the intended message for the indicator, and the target
audience before considering data availability. After identifying environmental indicators using
this approach, however, it is necessary to fully consider the availability and quality of data to
support the candidate indicators.
This section describes an approach for evaluating data to support environmental indicators.
The evaluation approach will vary depending on the goals and objectives of the indicator project
and the stringency of data requirements needed to achieve those goals. Therefore, the Indicator
Team, in consultation with its stakeholder group and data experts that could provide technical
insights, should determine project-specific data requirement:;. The Indicator Team may want to
identify and weight specific criteria to use in reviewing data sets.
Table 6-1, given at the end of this section, identifies criteria for evaluating the usefulness
of existing data in supporting the development of the final environmental indicators. The criteria
presented should be modified to best meet the needs of a particular indicator project.
In general, critical criteria for selecting data sets include the following:
• Availability of data on the selected parameters
* Appropriate temporal and spatial coverage
• Documented quality
• Accessibility.
Another critical criterion is that minimal standards of technical credibility, estimation
precision, and cost can be achieved by either the present data collection procedures or reasonable
modifications of them, because changes in data collection procedures might affect the technical
credibility, magnitude of the estimation error and associated sample size, and overall cost.
Revised Draft
32
May 1996
-------
Criteria for Selecting Environmental Indicators
It is possible that either the sampling procedures or laboratory analysis procedures will
change over the time that a data source is used to quantify an indicator and monitor progress.
These changes may.result from many factors, including advances in technology and changes in
budgets and uses of the data sets over time. The effect of these changes can be minimized by
using (I) measurements for which changes in technology are likely to improve the precision but
not affect the measurement bias and (2) procedures for which the measurement bias is relatively
insensitive to the magnitude of the collection effort. To the extent that this cannot be achieved,
a comparability study can be used to compare the indicator before and after the change. The
value of both the original and revised indicator can be used for some time to provide information
on how the two indicators compare. This same procedure can also be used if a entirely new data
set is used for the revised indictor.
Application of evaluation criteria to determine which data sets best support candidate
indicators is described in Step 2b of the selection process. After evaluation of potential data sets,
the selection of final indicators takes place. This is Step 3, the final step, of the indicator
selection process.
Environmental, indicators provide an accurate measure and an objective description of
current environmental trends and patterns. The process for selecting environmental indicators
described in this document facilitates the development of unbiased indicators supported by
existing data sets. The Indicator Team and stakeholder group should tailor the selection
methodology and criteria described within each section to fit the needs of specific indicator
projects. Causal frameworks, such as the PSR framework described in this document, provide
context and organization structure for environmental indicators. Project-specific factors such as
the intended audience, message, and use influence the presentation style of selected
environmental indicators. Environmental indicator development promotes effective information
collection, quantification, and communication, and illustrates the need for continued research in
this area.
Revised Draft . 33 Ma? 19%
-------
Process for Selecting Indicators
Table 6-1. Criteria for Selecting Existing Data Sets to Quantify Indicators
Criteria
'Data'Avai lability
•Appropriate
Temporal Coverage
•Appropriate
Spatial Coverage
Data Quality
•Critical criterion
Definition
Does the data set provide measurements of
the parameters) or variable(s) specified in
the indicator.
Are appropriate historical data available so
that a baseline and/or trends can be
established?
Do the data cover the area of interest?
Information should be available on a
national basis for a national program. If
the information is compiled from local or
regional data, can the information be
aggregated using scientifically and
statistically valid procedures?
Are the data of known quality (i.e., are
there (1) documented QA/QC procedures
for the collection, analysis, and
presentation of data, (2) documentation of
any deviations from the procedures, and
(3) quantitative information on both
sampling and non-sampling errors)?
Additional Considerations
Docs (he data set measure supporting
parameters, such as those needed for data
interpretation (e.g., pH for metals,
temperature for dissolved oxygen)?
Docs the data set provide all necessary
information to support the data (e.g.,
location, date, weather, tide level)?
Arc- data available for lime periods crucial
for data interpretation (e.g., dissolved oxygen
dati in the summer)?
Does temporal coverage within reporting
cycles (usually annually) have gaps? If gaps
exi.it, they should not exclude data that will
significantly affect the indicator.
An; the data representative (i.e., not focused
on "hot spots")?
Do the data provide sufficient coverage to
determine sources, cause, and effect (e.g..
can they separate pollution/contamination
from natural background)?
Do the data use accepted geographic
conventions?
An: the data of appropriate scale and detail?
Is information on field and laboratory
methods provided?
Ars detection limits provided, where
applicable?
Were results of accuracy checks provided
(e.g., duplicates, replicates, split samples,
spike recoveries, instrument calibration)?
W:re lab audits performed and reported?
W:re there statistical checks on the data,
including data entry procedures?
Were problems identified'in the data? If so.
how (e.g., using flags, leaving data points
missing, reporting zeros)?
Were assumptions and limitations of the data
discussed?
Was a point of contact provided?
Revised Draft
34
May 1996
-------
Criteria for Selecting Environmental Indicators
Table 6-1. Criteria for Selecting Existing Data Sets to Quantify Indicators (continued)
Criteria
Data Accessibility
Technical
Credibility
Acceptable
Estimation Error
Acceptable Cost
*Critical criterion
Definition
Are the data able to be analyzed using
existing data retrieval and analysis
procedures?
Did the procedures used to manage and
analyze the data follow accepted
professional practices. Are the sample and
data collection procedures consistent with
the use of the data as a measure of the
indicator, as judged by technical experts in
the field who are familiar with the data?
The calculated bias in the indicator should
be insensitive to the magnitude of the data
collection effort and to political pressures.
In general, this criterion will eliminate
self-reported data from consideration.
Is the precision and bias of the indicator
acceptable given the desired precision
specified by the program?
Is the cost of data collection, management,
and analysis within programmatic
guidelines?
Additional Considerations
Are data able to be used, or do
confidentiality concerns limit data access?
Are the data available in electronic format?
Are the appropriate computer software and
hardware technologies available to access the
data?
Are the data in an acceptable format?
Is there a point of contact available to
resolve issues?
Are the data consistent with that of similar st
udies and information?
Are the data results consistently interpreted?
Are the data sufficiently accurate to meet the
goals and objectives of the indicator project?
Can the indicator and supporting data be
reproduced, updated, and/or modified at an
acceptable cost?
Revised Draft
35
May 1996
-------
-------
APPENDIX A
PRESSURE-STATE-RESPONSE/EFFECTS FRAMEWORK
Revised Draft
May 1996
-------
-------
Appendix A
, EPA is considering an expanded version of the OECD PSR framework/5 This enhanced
conceptual framework adds "Effects" as a category to describe impacts of environmental change
on human health and welfare. The updated version, PSR/E, shown in Figure A-l, also divides
each category into subcategories (e.g., a distinction is made between direct pressures and
underlying pressures). In addition, U seeks to link the PSR framework explicitly to society's
environmental values, goals, and priorities. Moreover, the framework aims for the incorporation
of spatially referenced (geographic) information, organized on the basis of ecologically defined
geographic scales; the'adoption of sustainability targets; and the multiscaled use of information.
Table A-l provides a case study illustrating the proposed framework.
Indicators of effects under the PSR/E framework describe relationships between two or
more pressure, state, and/or response variables. They are based on models and analyses that
provide plausible evidence of a linkage between a problem, potential causes, and/or solutions.
The most important types of effects include effects of underlying pressures on human activities;
effects of human activities (indirect pressures) on levels of biophysical stressors (direct
pressures); and effects of pressures or responses on ecological state, human health, and human
welfare.26 Effect indicators are perhaps the most comprehensive environmental indicators
because they describe relationships among two or more variables within the other categories.
Theoretically, effect indicators should provide a greater degree of certainty in describing cause
and effect relationships than just pressure, state, or response indicators alone; however, the time
involved in data collection to develop effect indicators may detract from their usefulness as an
evaluation criterion of policy performance."
:!United States Environmental Protection Agency. Conceptual Framework to support Development and Use of
Environmental Information In Decision-Making, April, 1995.
• :6USEPA. op. cit.
"Ibid.
Revised Draft A-l May 1996
-------
Process for Selecting Indicators
i
2
V3
1
a-
i
1
i
2
Eh
I
s
2
w
ig
i
i
3
£
o*
Revised Draft
A-2
May 1996
-------
Appendix A
Table A-l. Case Study of Pressure, State, Response/Effects Framework and
Subcategories (adapted from USEPA, 1995)'
Example Environmental Value, Goal and Priority. Citizens desire a healthy and vibrant recreational fishery tn the nation's
estuaries : value). The goal is to reduce nutrient and toxic loadings from the adjacent watershed to increase current fish
populations. Reducing nutrient and toxic pollutant loadings from agricultural and urban areas are the top priority.
Pressures
(P)
Underlying Pressures
Sociotechnkal Forces:
population, technology,
social structure, attitudes &
practices, policies
(e.g., burgeoning
populations and increasing
density of development)
indirect Pressures Human
Activities:
agriculture, mining,
manufacturing, transport.
energy consumption
(e.g., runoff from
agriculture and urban
areas, direct discharges)
and
Natural Processes/Events
< volcanic eruptions,
wildfires)
(e.g., natural erosion)
Direct Pressures
Biophysical Stressors:
pollutants, resource
extraction, land use change,
exotic species
(e.g., animal wastes
discharges, urban/suburban
fertilizer runoff, failing
septic systems, increasing.
levels of impervious area)
State of the Environment
(S)
Global Ecosystem
Ambient conditions and
trends (chemical, physical,
bio/ecological); Status of
"valued environmental
attributes" (VEAs)
(e.g., large scale
eutrophication, fish kills,
declines in species diversity
or abundance, loss of
habitat)
Regional Scale Ecosystems
conditions and trends
(chemical, physical,
bio/ecological); Status of
"valued environmental
attributes" (VEAs)
(e.g., regional
eutrophication, fish lolls,
declines in species diversity
or abundance, loss of
habitat)
. Local Scale Ecosystems
conditions and trends
(chemical, physical.
bio/ecological); Status of
"valued environmental
attributes" (VEAs) .
(e.g., local eutrophication,
fish lolls, declines in
species diversity or
abundance, fish tissue
concentration of toxins)
Human Health & Welfare
conditions and trends
(chemical, physical,
bio/ecological): Status of
"valued environmental
attributes" (VEAs)
(e.g., concentrations of
toxins in humans, fishing
bans or consumption
advisories, incidence of
disease)
Societal Responses
(R)
Government Actions;
Legislation, regulations,
policies, monitoring,
enforcement actions,
investments, international
agreements, etc.
(e.g., implementation of
Clean Water Act)
Private Sector Activities
Compliance, waste
treatment, mitigation.
cleanups, process redesign,
etc.
(e.g., pollution prevention
: planning and
implementation, animal
waste management,
conservation tillage
integrated pest
management)
Individual/
Household Attitudes &
Actions
Recycling, conservation,
contribution to NGOs, etc.
(e.g., household hazardous
waste collection programs,
toxics use reduction, use of
integrated pest
management, reduced
fertilizer use)
Cooperative Efforts
Research. NGOs, public-
private partnerships, etc.
(e.g., development of
watershed management
plans, integrated water
quality and living resources
monitoring programs)
Effects (E)
(Relationships between P,
S and/or R)
Linkages
between levels of Pressures
(Underlying, indirect. &
Direct), or between
Pressures and Responses
(e.g., effects of population
growth on agricultural
and industrial output)
Ecological Effects
Relationships between
Direct Pressures or Societal
Responses and State of the
Environment
(e.g., increased density and
area! extent of submerged
aquatic vegetation, reduced
incidences of hypoxia)
Human Health Effects
of Direct Pressures,
Ecological Changes (in
State), or Societal
Responses
(e.g., increased number of
fishing bans and human
consumption advisories
increased disease)
Human Welfare Effects
of Ecological Changes (in
State), or Societal
Responses
(e.g, incidence of disease
attributed to fish or
shellfish consumption,
reduced economic value of
fishery)
Revised Draft
A-3
May 1996
-------
-------
APPENDIX B
DRAFT SHORT FORM FOR SCREENING CANDIDATE DATA SETS
Revised Draft May 1996
-------
-------
DittS*
Data S* Acronym/Short Vain*:
(•Short*) Form
Data Conpkud:
1.4
:.
l.i FulINam«ofD«nS«:
1.2 St
L3
2J How
(I
(1
(ctekOM)
-------
2.4 Overall response .rate: __%
2.5 Wtiat is the geographic coverage of the dot in die data sec? (cheek one)
r j National
[ ] Regional: Spediy:
(] San: Soedfr
2.6 WhM dme period (y««) (km the dan MI cover? From 19^^(0 19 __.
2.7 Is the data eoflecdoB a
2.1 Frequency with wfakb d» surrey or dan coflecdoa tflbit it npand:
3.1 Foe WOMB of tft§ fbflowBaj typoi of
[] WeV
(J Sot
[] Air
u
-------
•~~"* j:-^r~'~*'; , *>\*,vXmZ'•_ ~
4. i Did the following dan collection activities have written procedural fi«KM**»g
soadard mediods) and wort the procedures doamwnted through a QA/QC program
of
review?
Enviroooeooi
Biological sampling
miytis:
WntttB.
C^^^J
CJ
U
u
[I
[1
QA/QC review
II
C1
U
n
n
u
5.1 For lay
iaarvtior
-------
6.1
-------
APPENDIX C
SELECTION CRITERIA USED IN OTHER ENVIRONMENTAL
INDICATOR PROJECTS
Revised Draft May 1996
-------
-------
Appendix C
Chesapeake Bay Program. 1995. Summary of Environmental Indicator Workshops (draft
document):
Does the indicator reflect the message we want to communicate? Can the public relate?
Who is the intended audience?
Is the indicator tailored to the intended audience?
Do we have the data? Defensible and valid? Consensus on interpretation? If we do not have
the data, should we recommend its collection?
Is there a benchmark against which we can measure our progress?
Is it simple and direct?
Will it help to answer the question, "How is the Bay?"
Does it reflect established Chesapeake Bay Program goals?
Can we combine this indicator with others to form indices/multi-species "community" indicators?
Revised Draft C-l MaX l996
-------
Process for Selecting Indicators
The State Environmental Goals and Indicators Project. 1995. Prospective Indicators for
State L'se in Performance Agreements, Florida State University.
Essential Criteria
Measurable—The indicator measures a feature of the environment that can be quantified simply
using standard methodologies with known degree of accuracy and precision.
Data Quality—The data supporting the indicators are adequately supported by sound collection
methodologies, data management systems and quality assurance procedures to ensure that the
indicator is accurately represented. The data should be clearly defined, verifiable, scientifically
acceptable and easy to reproduce.
Importance—The indicator must measure some aspect of environmental quality that reflects an
issue of major national importance to states and to the fedend government in demonstrating the
current and future conditions of the environment.
Relevance—The indicator should be relevant to desired significant policy goal, legal mandate,
or agency mission (e.g., contaminated fish fillets for consumption advisories; species of
recreational or commercial value) that provides information of obvious value that can be easily
related to the public and decision makers.
Representative—Changes in the indicator are highly correlated to trends in the other parameters
or systems they are selected to represent.
Appropriate Scale—The indicator responds to changes on an appropriate geographic (e.g.,
national or regional) and/or temporal (e.g., yearly) scale.
Trends—The data for the indicator should have been collected over a sufficient period of time
to allow some analysis of trends or should provide a baseline for future trends. The indicator
should show reliability over time, bringing to light a representative trend, preferably annual.
Decision Support—The indicator should provide information to a level appropriate for making
policy.decisions. Highly specific and special parameters . useful to technical staff, will not be
of much significance to policy staff or management decision makers.
Preferable Criteria
Results—The indicator should measure a direct environmental result (e.g., an impact on human
health or ecological conditions). Indicators expressing changes in ambient conditions or changes
in measures reflecting discharges or releases are acceptable, but not preferred. Process measures
(e.g., permits, compliance and enforcement activities, etc.) sire not acceptable.
Understandable—The indicator should be simple and clear, and sufficiently nontechnical to be
comprehensible to the general public with brief explanation, The indicator should lend itself to
effective and appealing display and presentation.
Revised Draft
C-2
May 1996
-------
Appendix C
Sensitivity—The indicator is able to distinguish meaningful difference in environmental
conditions with an acceptable degree of resolution. Small changes in the indicator show
measurable results.
Integrates effects/exposures—The indicator integrates effects or exposures over time and space
and responds to the cumulative impacts of multiple stressors. It is broadly applicable to many
stressors and sites.
Data comparability—The data supporting an indicator can be compared to existing and past
measures of conditions to .develop trends and define variation.
Cost effective/availability—The information for an indicator is available or can be obtained with
reasonable cost and effort and provides maximum information per unit effort.
Anticipatory—The indicator is capable of providing an early warning of environmental change.
Revised Draft C-3 Ma? 1996
-------
Process for Selecting Indicators
Adriaanse, A. 1993. Environmental Policy Performance Indicators: A Study on the
Development of Indicators for Environmental Policy in the Netherlands.2*
Indicators should be as aggregative as possible.
They must have a definite appeal, partly by being aptly presented.
They must reflect a trend, with a time scale that is tailored to the problems.
They must relate to cause and effect, or in other words, to the causal chain.
The course of actual developments in time must be seen in relation to existing policy objectives
and necessary measures.
They must be verifiable and reproducible/
Further refinement of the above resulted in the following criteria:
Acceptable Quality for Data and Methodology (e.g., clearly defined, accurately described, socially
and scientifically acceptable, easy to reproduce)
Sensitivity in time
Policy relevance
Recognizability and clarity
28Adriaanse, A. 1993. Environmental Policy Performance Indicators.
Revised Draft
C-4
May 1996
-------
Appendix C
The Organization for Economic Cooperation and Development. 1994. Environmental
Indicators.
Policy Relevance and Utility for Users
Provide a representative picture of environmental conditions, pressures on the environment, or
society's responses.
Be simple, easy to interpret, and able to show trends over time.
Be responsive to changes in the environment and related human activities.
Provide a basis for international comparisons.
Be either national in scope or applicable to regional environmental issues of national significance.
Have a threshold or reference value against which to compare it so that users are able to assess
the significance of the values associated with it.
Analytical Soundness
Be theoretically well founded in technical and scientific terms.
Be based on international standards and the international consensus about the validity.
Lend itself to being linked to economic models, forecasting information systems.
Measurability
Readily available or made available at a reasonable cost/benefit ratio.
Adequately documented and of known quality.
Updated at regular intervals in accordance with reliable procedures.
Revised Draft C-5 Ma? l996
-------
Process for Selecting Indicators
Interagency Working Group for Sustainable Development Indicators. 1995. Criteria and
Plan for Selection of Indicators of Sustainable Development and Sustainability (draft
document):
Understandable—Should not be obscure or statistically difficult to understand. Clear,
understandable, sufficiently universal to be easily communicated.
Relatable to sustainable development and sustainability
Have a constant definition over time. Definition, measurement technique, and analytical
methodology is constant over time. Must avoid.discontinuities.
Sufficient historical data available. Preferable to have a recDrd for a 20- to 50-year time period.
Available in electronic form.
Indicators should be national in scope, including summary data and information that scales and
is available at the state, regional, and local levels.
Quality known—Metadata should be included for all indicators that shows the quality. This data
should include such information as sensitivity, uncertainty, variability, precision, accuracy, error
and similar analyses.
If combining information, should include economic, environmental, and social information.
If building toward an index, indicators should be able to t>; combined.
Relevance to policy and issues of concern.
Revised Draft
C-6
May 1.996
-------
Appendix C
QLF/Atlantic Center for the Environment. 1995. Criteria and Ranking Scheme for
Indicators of Sustainability:
A true measure of sustainability.
Understandable to the community at large.
Focused on a long-term view of the community (20 to 50+ years)
A link between different aspects of the community (social, economic, environmental)
A yard stick against which to measure potential places
A measure of community level sustainability that was not at the expense of global sustainability
Based on reliable, easily available information
Available on a regular (hopefully yearly or biennially) basis
Revised Draft C-7 Ma* 19%
-------
U.S. EPA Headquarters Library
Mail code 3201
1200 Pennsylvania Avenue NW
v Washington DC 20460
------- |