United States
Environmental
Protection Agency
Office of Water
(4503F)
Washington, DC
EPA 841-B-97-OD9
July 1997
svEPA
Techniques for Tracking,
Evaluating, and Reporting
The Implementation of Nonpoint
Source Control Measures
Forestry
-------
-------
TECHNIQUES FOR TRACKING, EVALUATING,
AND REPORTING THE IMPLEMENTATION
OF NONPOINT SOURCE CONTROL
MEASURES
II. FORESTRY
Final
July 1997
Prepared for
Steve Dressing
Nonpoint Source Pollution Control Branch
United States Environmental Protection Agency
Prepared by
Tetra Tech, Inc.
EPA Contract No. 68-C3-0303
Work Assignment No. 4-51
-------
-------
TABLE OF CONTENTS
Chapter 1 Introduction
1.1 Purpose of Guidance 1-1
1.2 Background 1-1
1.3 Types of Monitoring 1-3
1.4 Quality Assurance and Quality Control 1-5
1.5 Data Management 1-5
Chapter 2 Sampling Design
2.1 Introduction 2-1
2.1.1 Study Objectives 2-1
2.1.2 Probabilistic Sampling 2-2
2.1.3 Measurement and Sampling Errors 2-8
2.1.4 Estimation and Hypothesis Testing 2-10
2.2 Sampling Considerations 2-13
2.2.1 Site Selection 2-13
2.2.2 Data to Support Site Selection 2-14
2.2.3 Example State and Federal Programs 2-14
2.3 Sample Size Considerations 2-17
2.3.1 Simple Random Sampling 2-19
2.3.2 Stratified Random Sampling 2-23
2.3.3 Cluster Sampling 2-26
2.3.4 Systematic Sampling 2-26
Chapter 3 Methods for Evaluating Data
3.1 Introduction 3-1
3.2 Comparing the Means from Two Independent Random Samples 3-2
3.3 Comparing the Proportions from Two Independent Samples 3-3
3.4 Comparing More Than Two Independent Random Samples 3-4
3.5 Comparing Categorical Data 3-4
Chapter 4 Conducting the Evaluation
4.1 Introduction 4-1
4.2 Choice of Variables 4-2
4.3 Expert Evaluations 4-4
4.3.1 Site Evaluations 4-4
4.3.2 Rating Implementation of Management Measures and Best
Management Practices . . 4-9
4.3.3 Rating Terms 4-11
4.3.4 Consistency Issues 4-12
4.3.5 Postevaluations Onsite Activities 4-13
-------
Table of Contents
4.4 Self-Evaluations 4-13
4.4.1 Methods 4-13
4.4.2 Cost 4-14
4.4.3 Questionnaire Design 4-15
4.5 Aerial Photography 4-17
Chapter 5 Presentation of Evaluation Results
5.1 Introduction 5-1
5.2 Audience Identification 5-2
5.3 Presentation Format 5-2
5.3.1 Written Presentations 5-3
5.3.2 Oral Presentations 5-3
5.4 For Further Information 5-6
References R-l
Glossary G-l
Appendix A: Statistical Tables A-l
Appendix B: Sample Evaluation Forms B-l
Index M
-------
List of Tables
Table 2-1 Applications of four sampling designs for implementation
monitoring 2-3
Table 2-2 Errors in hypothesis testing 2-12
Table 2-3 Definitions used hi sample size calculation equations . 2-18
Table 2-4 Comparison of sample size as a function of various parameters 2-20
Table 2-5 Common values of (Za + Z2P) for estimating sample size 2-23
Table 2-6 Allocation of Samples 2-25
Table 2-7 Number of harvest sites implementing recommended BMPs 2-27
Table 3-1 Contingency table of implemented BMP and rating of
installation and maintenance 3-5
Table 3-2 Contingency table of expected harvest site type and
implemented BMP 3-6
Table 3-3 Contingency table of implemented BMP and rating of
installation and maintenance 3-7
Table 3-4 Contingency table of implemented BMP and sample year 3-8
Table 4-1 General types of information obtainable with self-evaluations
and expert evaluations 4-3
Table 4-2 Examples of variables related to management measure
implementation 4-6
List of Figures
Figure 2-1 Simple random sampling from a list and a map 2-4
Figure 2-2 Stratified random sampling from a list and a map 2-6
Figure 2-3 Cluster sampling from a list and a map 2-7
Figure 2-4 Systematic sampling from a list and a map 2-9
Figure 2-5 Graphical presentation of the relationship between bias,
precision, and accuracy 2-11
Figure 4-1 Potential variables and examples of implementation
standards and specifications 4-5
Figure 5-1 Timberland area by stand size class, East and West 5-4
Figure 5-2 Forest type groups on unreserved forest land in the East 5-5
Figure 5-3 Example written presentation slide 5-5
-------
-------
CHAPTER 1. INTRODUCTION
1.1 PURPOSE OF GUIDANCE
This guidance is intended to assist state,
regional, and local environmental
professionals in tracking the
implementation of best management
practices (BMPs) used to control nonpoint
source pollution generated by forestry
practices. Information is provided on
methods for sample site selection, sample
size estimation, sampling, and result
evaluation and presentation. The focus of
the guidance is on the statistical approaches
needed to properly collect and analyze data
that are accurate and defensible. A
properly designed BMP implementation
monitoring program can save both time and
money. For example, in 1993 forestry
operators notified the State of Idaho of
5,890 forestry operations (Colla, 1994).
The cost of determining the status of BMP
implementation on each of those forestry
operations would have exceeded the amount
budgeted, and thus statistical sampling of
sites was needed. This document provides
guidance for sampling representative
forestry operations to yield summary
statistics at a fraction of the cost of a
comprehensive inventory.
Some forestry nonpoint source projects and
programs combine BMP implementation
monitoring with water quality monitoring
to evaluate the effectiveness of BMPs in
protecting water quality (Curtis et al.,
1990; Rashin et al., 1994; USEPA,
1993b). For this type of monitoring to be
successful, the scale of the project usually
must be small (e.g., a watershed of a few
hundred to a few thousand acres).
The focus of this guide is on the design of
monitoring programs to assess forestry
management measure and best management
practice implementation, -with particular
emphasis on statistical considerations.
Accurate records of all the sources of
pollutants of concern and a census of how
all BMPs are operating are very important
for this type of monitoring effort.
Otherwise, it can be extremely difficult to
correlate BMP implementation with
changes in stream water quality. This
guidance does not address monitoring the
implementation and effectiveness of all
BMPs in a watershed. This guidance does
provide information to help program
managers gather statistically valid
information to assess implementation of
BMPs on a more general (e.g., statewide)
basis. The benefits of implementation
monitoring are presented in Section 1.3.
1.2 BACKGROUND
Pollution from nonpoint sources—sediment
deposition, erosion, nutrients, contaminated
runoff, hydrologic modifications that
degrade water quality, and other diffuse
sources of water pollution—is the largest
cause of water quality impairment in the
United States (USEPA, 1995). Congress
passed the Coastal Zone Act
Reaumorization Amendments of 1990
(CZARA) to help address nonpoint source
pollution in coastal waters. CZARA
-------
Introduction
Chapter 1
provides that each state with an approved
coastal zone management program develop
and submit to the U.S. Environmental
Protection Agency (EPA) and National
Oceanic and Atmospheric Administration
(NOAA) a Coastal Nonpoint Pollution
Control Program (CNPCP). State
programs must "provide for the
implementation" of management measures
in conformity with the EPA Guidance
Specifying Management Measures For
Sources Of Nonpoint Pollution In Coastal
Waters, developed pursuant to Section
6217(g) of CZARA (USEPA, 1993a).
Management measures (MMs), as defined
in CZARA, are economically achievable
measures to control the addition of
pollutants to coastal waters, which reflect
the greatest degree of pollutant reduction
achievable through the application of the
best available nonpoint pollution control
practices, technologies, processes, siting
criteria, operating methods, or other
alternatives. Many of EPA1 s MMs are
combinations of BMPs. For example,
depending on site characteristics,
implementation of the Road Management
MM might involve use of the following
BMPs: installing or regrading water bars;
clearing road inlet and outlet ditches, catch
basins, culverts, and road-crossing
structures of obstructions; revegetating
road surfaces; and inspecting closed roads.
CZARA does not specifically require that
states monitor the implementation of MMs
and BMPs as part of their CNPCPs. State
CNPCPs must, however, provide for
technical assistance to local governments
and the public for implementing the MMs
and BMPs. Section 6217(b) states:
Each State program . . . shall
provide for the implementation, at a
minimum, of management measures
. . . and shall also contain . . .
(4) The provision of technical and
other assistance to local
governments and the public for
implementing the measures . . .
which may include assistance ... to
predict and assess the effectiveness
of such measures ....
EPA and NOAA also have some
responsibility under Section 6217 for
providing technical assistance to implement
state CNPCPs. Section 6217(d), Technical
assistance, states:
[NOAA and EPA] shall provide
technical assistance ... in
developing and implementing
programs. Such assistance shall
include: . . .
(4) methods to predict and assess the
effects of coastal land use
management measures on coastal
water quality and designated uses.
This guidance document was developed to
provide technical assistance as described in
CZARA Sections 6217(b)(4) and 6217(d),
but the techniques described can be used
for other similar programs and projects.
For instance, monitoring projects funded
under Clean Water Act (CWA) Section
319(h) grants, efforts to implement total
maximum daily loads developed under
CWA Section 303(d), storm water
permitting programs, and other programs
could benefit from knowledge of BMP
implementation.
-------
Chapter 1
Methods to assess the implementation of
MMs and BMPs, then, are a key focus of
the technical assistance to be provided by
EPA and NOAA. Implementation
assessments can be done on several scales.
Site-specific assessments can be used to
assess individual BMPs or MMs, and
watershed assessments can be used to look
at the cumulative effects of implementing
multiple MMs. With regard to "site-
specific" assessments, individual BMPs
must be assessed at the appropriate scale
for the BMP of interest. For example, to
assess the implementation of MMs and
BMPs for forest roads at harvest sites, only
the roads at timber harvesting sites would
need to be inspected. In this example, the
scale would be a timber harvest area and
the sites would be active and inactive roads
at the harvest areas. To assess MM and
BMP implementation at streamside
management areas (SMAs), the proper
scale might be a harvest area larger than 10
acres and the sites could be areas
encompassed by buffer areas for 200-meter
stretches of stream. For site preparation
and forest regeneration, the scale and site
might be an entire harvest site. Site-
specific measurements can then be used to
extrapolate to a watershed or statewide
assessment. It is recognized that some
studies might require a complete inventory
of MM and BMP implementation across an
entire watershed or other geographic area.
1.3 TYPES OF MONITORING
The term monitor is defined as "to check or
evaluate something on a constant or regular
basis" (Academic Press, 1992). It is
possible to distinguish among various types
of monitoring. Two types, implementation
and trend (i.e., trends in implementation)
monitoring, are the focus of this guidance.
These types of monitoring can be used to
address the following goals:
• Determine the extent to which MMs
and BMPs are being implemented hi
accordance with design standards
and specifications.
• Determine whether there has been a
change in the extent to which MMs
and BMPs are being implemented.
In general, implementation monitoring is
used to determine whether goals,
objectives, standards, and management
practices are being implemented as detailed
in implementation plans. In the context of
BMPs within state CNPCPs,
implementation monitoring is used to
determine the degree to which MMs and
BMPs required or recommended by the
CNPCPs are being implemented. If
CNPCPs call for voluntary implementation
of MMs and BMPs, implementation
monitoring can be used to determine the
success of the voluntary program (1) within
a given monitoring period (e.g., 1 or 2
years); (2) during several monitoring
periods, to determine any temporal trends
in BMP implementation; or (3) in various
regions of the state.
Trend monitoring involves long-term
monitoring of changes in one or more
parameters. As discussed in this guidance,
public attitudes, land use, or the use of
different forestry practices are examples of
parameters that could be measured with
-------
Introdudion
Chapter 1
trend monitoring. For example, the State
of Idaho, Department of Lands, tracks
tre"nds in the number of forestry operations
and enforcement actions (Colla, 1994). In
addition, to isolate the impacts of MMs or
BMPs on water quality, it is necessary to
track their implementation over time.
Because trend monitoring involves
measuring a change (or lack thereof) in
some parameter over time, it is necessarily
of longer duration than implementation
monitoring and requires that a baseline, or
starting point, be established. Any changes
in the measured parameter are then detected
in reference to the baseline.
Implementation and the related trend
monitoring can be used to determine (1)
which MMs and BMPs are being
implemented, (2) whether MMs and BMPs
are being implemented as designed, and (3)
the need for increased efforts to promote or
induce use of MMs and BMPs. Data from
implementation monitoring, used in
combination with other types of data (e.g.,
water quality data), can be useful in
meeting a variety of other objectives,
including the following (Hook et al., 1991;
IDDHW, 1993; Schultz, 1992):
• To evaluate BMP effectiveness for
protecting soil and water resources.
• To identify areas in need of further
investigation.
• To establish a reference point of
overall compliance with BMPs.
• To determine whether landowners/
forestry operators are aware of
BMPs.
• To determine whether landowners/
forestry operators are using the
advice of forestry BMP experts.
• To identify any BMP
implementation problems specific to
a land ownership category.
• To evaluate whether any forestry
practices cause environmental
damage.
• To compare the effectiveness of
alternative BMPs.
MacDonald et al. (1991) describe
additional types of monitoring, including
effectiveness monitoring, baseline
monitoring, project monitoring, validation
monitoring, and compliance monitoring.
As emphasized by MacDonald and others,
these monitoring types are not mutually
exclusive and the distinction between them
is usually determined by the purpose of the
monitoring.
Effectiveness monitoring is used to
determine whether MMs or BMPs, as
designed and implemented, are effective in
meeting management goals and objectives.
Effectiveness monitoring is a logical
follow-up to implementation monitoring. It
is essential that effectiveness monitoring
include an assessment of the adequacy of
the design and installation of MMs and
BMPs. For instance, the objective of
effectiveness monitoring could be to
-------
Chapter 1
evaluate the effectiveness of MMs and
BMPs as designed and installed, or to
evaluate the effectiveness of MMs and
BMPs that are designed and installed
adequately or to standards and
specifications. Effectiveness monitoring is
the subject of another EPA guidance
document, Nonpoint Source Monitoring
and Evaluation Guide (USEPA, 1996).
Effectiveness monitoring for forestry BMPs
is also addressed in a U.S. Forest Service
document, Evaluating the effectiveness of
forestry best management practices in
meeting water quality goals or standards
(Dissmeyer, 1994).
1.4 QUALITY ASSURANCE AND QUALITY
CONTROL
An integral part of the design phase of any
nonpoint source pollution monitoring
project is quality assurance and quality
control (QA/QC). Development of a
quality assurance project plan (QAPP) is
the first step of incorporating QA/QC into
a monitoring project. The QAPP is a
critical document for the data collection
effort inasmuch as it integrates the technical
and quality aspects of the planning,
implementation, and assessment phases of
the project. The QAPP documents how
QA/QC elements will be implemented
throughout a project's life. It contains
statements about the expectations and
requirements of those for whom the data is
being collected (i.e., the decision maker)
and provides details on project-specific data
collection and data management procedures
that are designed to ensure that these
requirements are met. Development and
implementation of a QA/QC program,
including preparation of a QAPP, can
require up to 10 to 20 percent of project
resources (Cross-Smiecinski and
Stetzenback, 1994), but this cost is
recaptured in lower overall costs due to the
project being well planned and executed.
A thorough discussion of QA/QC is
provided in Chapter 5 of EPA's Nonpoint
Source Monitoring and Evaluation Guide
(USEPA, 1996).
1.5 DATA MANAGEMENT
Data management is a key component of a
successful MM or BMP implementation
monitoring effort. The data management
system that is used—which includes the
quality control and quality assurance
aspects of data handling, how and where
data are stored, and who manages the
stored data—determines the reliability,
longevity, and accessibility of the data.
Provided that the data collection effort was
planned and executed well, an organized
and efficient data management system will
ensure that the data can be used with
confidence by those who must make
decisions based upon it, the data will be
useful as a baseline for similar data
collection efforts in the future, the data will
not become obsolete (or be misplaced!)
quickly, and the data will be available to a
variety of users for a variety of
applications.
Serious consideration is often not given to a
data management system prior to a data
collection effort, which is precisely why it
is so important to recognize the long-term
value of a small investment of time and
-------
Introduction
Chapter 1
money in proper data management. Data
management competes with other agency
priorities for money, staff, and time, and if
the importance and long-term value of
proper data management is recognized
early in a project's development, the more
likely it will be to receive sufficient
funding. Overall, data management might
account for only a small portion of a
project's total budget, but the return on the
investment is great when it is considered
that the larger investment in data collection
can be rendered virtually useless unless
data is managed adequately.
Two important aspects of data that should
be considered when planning the initial data
collection effort and a data management
system are data life cycle and data
accessibility. The data life cycle can be
characterized by the following stages: (1)
Data is collected; (2) data is checked for
quality; (3) data is entered into a data base;
(4) data is used, and (5) data eventually
becomes obsolete. The expected usefulness
and life span of the data should be
considered during the initial stages of
planning a data collection effort, when the
money, staff, and tune that are devoted to
data collection must be weighed against its
usefulness and longevity. Data with a
limited use and that is likely to become
obsolete soon after it is collected is a
poorer investment decision than data with
multiple applications and a long life span.
If a data collection effort involves the
collection of data of limited use and a short
life span, it might be necessary to modify
the data collection effort—either by
changing its goals and objectives or by
adding new ones—to increase the breadth
and length of the data's applicability. A
good data management system will ensure
that any data that are collected will be
useful for the greatest number of
applications for the longest possible time.
Data accessibility is a critical factor in
determining its usefulness. Data attains its
highest value if it is as widely accessible as
possible, if access to it requires the least
amount of staff effort as possible, and if it
can be used by others conveniently. If data
are stored where those who might need it
can obtain it with little assistance, it is
more likely to be shared and used. The
format for data storage determines how
conveniently the data can be used.
Electronic storage in a widely available and
used data storage format makes it
convenient to use. Storage as only a paper
copy buried in a report, where any analysis
requires entry into an electronic format or
time-consuming manipulation, makes data
extremely inconvenient to use and unlikely
that it will be used.
The following should be considered for the
development of a data management
strategy:
• What level of quality control should the
data be subject to? Data that will be
used for a variety of purposes or that
will be used for important decisions
should receive a careful quality control
check.
• Where and how will the data be stored?
The options for data storage range from
a printed final report on a bookshelf to
an electronic data base accessible to
-------
Chapter 1
government agencies and the public.
Determining where and how data will
be stored therefore also requires careful
consideration of the question: How
accessible should the data be?
Who will maintain the data base? Data
stored in a large data base might be
managed by a professional data
manager, while data kept in agency files
might managed by people with various
backgrounds over the course of time.
How much will data management cost?
As with all other aspects of a data
collection effort, data management costs
money and this cost must be balanced
with all other costs involved in the
project.
-------
-------
CHAPTER 2. SAMPLING DESIGN
2.1. INTRODUCTION
This chapter discusses recommended
methods for designing sampling programs to
track and evaluate the implementation of
nonpoint source control measures. This
chapter does not address whether the
management measures (MMs) or best
management practices (BMPs) are effective
since no water quality sampling is done.
Because of the variation in forestry practices
and related nonpoint source control measures
implemented throughout the United States,
the approaches taken by various states to
track and evaluate nonpoint source control
measure implementation will differ.
Nevertheless, all approaches should be based
on sound statistical methods for selecting
sampling strategies, computing sample sizes,
and evaluating data. EPA recommends that
states consult with a trained statistician to be
certain that the approach, design, and
assumptions are appropriate to the task at
hand.
As described in Chapter 1, implementation
monitoring is the focus of this guidance.
Effectiveness monitoring is the focus of
another guidance prepared by EPA, the
Nonpoint Source Monitoring and Evaluation
Guide (USEPA, 1996). Dissmeyer (1994)
also provides substantial information
regarding QA/QC, statistical considerations,
BMP effectiveness monitoring, and
monitoring methods. The recommendations
and examples in this chapter address two
primary monitoring goals:
• Determine the extent to which MMs
and BMPs are implemented in
accordance with design standards and
specifications.
• Determine whether there has been a
change in the extent to which MMs
and BMPs are being implemented.
For example, State forestry agencies might
be interested hi whether streamside
management areas (SMAs) at harvest sites
associated with all types of forest ownerships
(industrial, private nonindustrial, federal,
and state) are in compliance with design
standards. State forestry agencies might also
be interested in the percentage of owners of
nonindustrial private forest land that are
correctly implementing the BMPs specified
in a voluntary implementation program.
2.1.1. Study Objectives
To develop a study design, clear,
quantitative monitoring objectives must be
developed. For example, the objective might
be to estimate to within ±5 percent the
percent of harvest sites that have adequate
SMAs. Or perhaps a state is getting ready to
implement new administrative procedures to
ensure that purchasers of timber have been
advised of applicable BMPs. In this case,
detecting a 10 percent change in the number
of operators that implement the BMPs
specified in the timber sale contract might be
of interest. In the first example, summary
statistics are developed to describe the
current status, whereas in the second
example, some sort of statistical analysis
(hypothesis testing) is performed to
-------
Sampling Design
Chapter 2
determine whether a significant change has
really occurred. This choice has an impact
on how the data are collected. As an
example, summary statistics might require
unbalanced sample allocations to account for
variability such as site size, type, and
ownership, whereas balanced designs (e.g.,
two sets of data with the same number of
observations hi each set) are more typical for
hypothesis testing.
2.1.2. Probabilistic Sampling
Most study designs that are appropriate for
tracking and evaluating implementation are
based on a probabilistic approach since
tracking every operator is not cost-effective.
In a probabilistic approach, individuals are
randomly selected from the entire group.
The selected individuals are evaluated, and
the results provide an unbiased assessment of
the entire group. Applying the results from
randomly selected individuals to the entire
group is statistical inference. Statistical
inference enables one to determine, for
example, the probable percentage of timber
sales with adequate SMAs without visiting
every tract of land. One could also
determine whether the change hi timber sales
with appropriate streamside management is
within the range of what could occur by
chance or the change is large enough to
indicate a real modification of operator
practices.
The group about which inferences are made
is the population or target population, which
consists of population units. The sample
population is the set of population units that
are directly available for measurement. For
example, if the objective is to determine the
degree to which adequate SMAs have been
established, silvicultural operations for
which SMAs are an appropriate BMP (e.g.,
timber sales with nearby streams) would be
the sample population. Statistical inferences
can be made only about the target population
available for sampling. For example, if
implementation of erosion control is being
assessed and only public lands can be
sampled, inferences cannot be made about
the management of private lands.
The most common types of probabilistic
sampling that can be used for implementation
monitoring are summarized in Table 2-1. In
general, probabilistic approaches are
preferred. However, there might be
circumstances under which targeted sampling
should be used. Targeted sampling refers to
using best professional judgement for
selecting sample locations. For example,
state foresters deciding to evaluate all timber
sales in a given watershed would be targeted
sampling. The choice of a sampling plan
depends on study objectives, patterns of
variability in the target population, cost-
effectiveness of alternative plans, types of
measurements to be made, and convenience
(Gilbert, 1987).
Simple random sampling is the most basic
type of sampling. Each unit of the target
population has an equal chance of being
selected. This type of sampling is
appropriate when there are no major trends,
cycles, or patterns in the target population
(Cochran, 1977). Random sampling can be
applied in a variety of ways including
operator or timber sale selection. Random
samples can also be taken at different times
at a single site. Figure 2-1 provides an
-------
Chapter 2
Table 2-1. Applications of four sampling designs for implementation monitoring.
Samolina Desian
Simple Random
Sampling
Stratified Random
Sampling
Cluster Sampling
Systematic Sampling
.Comment „,,,-••,
Each population unit has an equal probability of being selected.
Useful when a sample population can be broken down into groups, or strata,
that are internally more homogeneous than the entire sample population.
Random samples are taken from each stratum although the probability of
being selected might vary from stratum to stratum depending on cost and
variability.
Useful when there are a number of methods for defining population units and
when individual units are clumped together. In this case, clusters are
randomly selected and every unit in the cluster is measured.
This sampling has a random starting point with each subsequent observation
a fixed interval (space or time) from the previous observation.
example of simple random sampling from a
listing of harvest sites and from a map.
If the pattern of MM and BMP
implementation is expected to be uniform
across the state, simple random sampling is
appropriate to estimate the extent of
implementation. If, however,
implementation is homogeneous only within
certain categories (e.g., federal, state, or
private lands), stratified random sampling
should be used.
In stratified random sampling, the target
population is divided into groups called
strata for the purpose of obtaining a better
estimate of the mean or total for the entire
population. Simple random sampling is then
used within each stratum. Stratification
involves the use of categorical variables to
group observations into more units, thereby
reducing the variability of observations
within each unit. For example, in a state
with federal, state, and private forests, there
might be different patterns of BMP
implementation. Lands in the state could be
divided into federal, state, and private as
separate strata from which samples would be
taken. In general, a larger number of
samples should be taken hi a stratum if the
stratum is more variable, larger, or less
costly to sample than other strata. For
example, if BMP implementation is more
variable on private lands, a greater number
of sampling sites might be needed in that
stratum to increase the precision of the
overall estimate. Cochran (1977) found that
stratified random sampling provides a better
estimate of the mean for a population with a
trend, followed hi order by systematic
sampling (discussed later) and simple
random sampling. He also noted that
stratification typically results in a smaller
variance for the estimated mean or, total than
that which results from comparable simple
random sampling.
-------
Sampling Design
Chapter 2
Harvest Site No.
1
2
3
4
5
6
7
8
9
10
11
12
• • •
142
143
144
145
146
147
148
149
150
Water Type
Stream
Stream
Lake/Pond
Stream
Lake/Pond
Lake/Pond
Lake/Pond
Stream
Stream
Lake/Pond
Stream
Stream
« • •
Lake/Pond
Lake/Pond
Stream
Lake/Pond
Stream
Stream
Lake/Pond
Stream
Lake/Pond
Ownership
Industry
Private Non-industrial ; ,
Industry
Industry
Private Non-industrial
Private Non-industrial
Industry
State
Federal
Private Non-industrial
Industry
State
• • •
: ;Jndustrv
State
State
Industry
Private Non-industrial
Private Non-industrial ,
Industry
Industry;
Federal
County Code '
14
•* -,;;•:--.;::• m^a^»
12
11
7 ' ',"
4
7
10
10
5
3
6
« • •
11
7
15
ts
&
9 '
1
12
11
Figure 2-1 a. Simple random sampling from a listing of harvest sites. In this listing, all harvest
sites are presented as a single list and sites are selected randomly from the entire list. Shaded
harvest sites represent those selected for sampling.
O
Figure 2-1 b. Simple random sampling from a map.
Dots represent harvest sites. All harvest sites of
interest are represented on the map, and the sites
to be sampled (open dots—O) were selected
randomly from all harvest sites on the map. The
shaded lines on the map could represent county,
watershed, hydrologic, or some other boundary, but
they are ignored for the purposes of simple random
sampling.
-------
Chapter 2
If the state believes that there will be a
difference between two or more subsets of
the sites, such as between types of ownership
or region, the sites can first be stratified into
these subsets and a random sample taken
within each subset (McNew, 1990). States
with silviculture implementation monitoring
programs commonly divide the sites by
ownership and county and/or region before
selecting survey sites. The goal of
stratification is to increase the accuracy of
the estimated mean values over what could
have been obtained using simple random
sampling of the entire population. The
method makes use of prior information to
divide the target population into subgroups
that are internally homogeneous. There are
a number of ways to "select" sites, or sets of
sites, to be certain that important information
will not be lost, or that MM or BMP use will
not be misrepresented as a result of treating
all potential survey sites as equal. Figure
2-2 provides an example of stratified random
sampling from a listing of harvest sites and
from a map.
Where data are available, it might be useful
to compare the relative percentages of
harvested timberland that is classified as
having high, medium, and low erosion
potentials. In cases where sediment is
impacting water quality, highly credible land
might be responsible for a larger share of
sediment delivery and would therefore be an
important target for tracking the
implementation of erosion controls. A
stratified random sampling procedure could
be used to estimate the percentage of total
harvested timberland with different erosion
potentials that have erosion controls in place.
For other water quality problems (e.g.,
spawning habitat in decline), other
stratification parameters (e.g., stream
classification) might be more appropriate.
Cluster sampling is applied in cases where it
is more practical to measure randomly
selected groups of individual units than to
measure randomly selected individual units
(Gilbert, 1987). In cluster sampling, the
total population is divided into a number of
relatively small subdivisions, or clusters, and
then some of the subdivisions are randomly
selected for sampling. In one-stage cluster
sampling, the selected clusters are sampled
totally. In two-stage cluster sampling,
random sampling is performed within each
cluster (Gaugush, 1987). For example, this
approach might be useful if a state wants to
estimate the proportion of harvest sites that
are following state-approved MMs or BMPs.
All harvest sites in a particular county can be
regarded as a single cluster. Once all
clusters have been identified, specific
clusters can be randomly chosen for
sampling. Freund (1973) notes that
estimates based on cluster sampling are
generally not as good as those based on
simple random samples, but they are more
cost-effective. As a result, Gaugush (1987)
believes that the difficulty associated with
analyzing cluster samples is compensated for
by the reduced sampling requirements and
cost. Figure 2-3 provides an example of
cluster sampling from a listing of harvest
sites and from a map.
Systematic sampling is used extensively in
water quality monitoring programs because it
is relatively easy to do from a management
perspective. In systematic sampling the first
sample has a random starting point and each
-------
Sampling Design
Chapter 2
I
1
1
1
1
1
1
1
1 Harvest Site No,
1
I 3
! 4
* * *
148
149
2
5
6
• • •
146
147
9
21
• • •
150
8
12
* • *
143
144
Water Type
Stream
t,ake/Pon
-------
Chapter 2
148
• ' '2
t - ~ W -
6
„> 10 x
12
5
7
143
s 146 ; ,
,/ ' 147*
8
9
'* 4 > <'
t1 , 142 " "
* • 150
3
149
1
«144 -
-P145 ' ,
Yypte
Lake/Pond
'' Stream- *
' , •? Stream ^
.ake/Pond
* fakeyPond
Stream
.ake/Pond
_ake/Pond
_ake/Pond
Stream
««• Stream „«. '*"'
Stream
Stream
Stream
'•* , Lake/Pond „ <^"
„ Lake/Pond
Lake/Pond
Stream
Stream
' •* Stream - •-
'LakeVPond
?"-?"*' Q^ner^Hip " t?'
Industry
^A- Private Nbn-industrial x
tndflstrv ^
Private Non-industrial
. Private Non-industrtaL
State
Private Non-industrial
Industry
State
&s Private Non-industrial f/
Private Non-industrial*
State
Federal
Industry
--*•"' Indu'stry
Federal^
Industry
Industry
Industry
- - * State
Industry
County Code
1
.3 "
-. 3
4
/» 5 's*
6
7
7
7
8' >-,'''
, ' ; 9 ' \
10
10
X ' ^1f
*11 ^ - ,
11
12
12
14
, - 15
15 /
Figure 2-3a. One-stage cluster sampling from a listing of harvest sites. Within the listing,
harvest sties were subdivided by the county in which they were located. Some of these counties
were then randomly selected, and all harvest sites within the selected counties were chosen for
sampling. Shaded harvest sites represent those located in the counties selected (i.e., counties
3,5,8,9, 11, and 15).
Figure 2-3b. Cluster sampling from a map. All
harvest sites in the area of interest are represented
on the map (closed {•} and open {O} dots). The
shaded lines on the map represent county
boundaries. Some of the counties were randomly
selected, and all harvest sites within those counties
(open dots - O) were selected for sampling. Some
other type of boundary, such as soil type or
watershed, could have been used to separate the
harvest sites for the sampling process.
-------
Sampling Design
Chapter 2
subsequent sample has a constant distance
from the previous sample. For example, if a
sample size of 70 is desired from a mailing
list of 700 operators, the first sample would
be randomly selected from among the first
10 people, say the seventh person.
Subsequent samples would then be based on
the 17th, 27th, ..., 697th person. In
comparison, a stratified random sampling
approach might be to sort the mailing list by
county and then to randomly select operators
from each county. Figure 2-4 provides an
example of systematic sampling from a
listing of harvest sites and from a map.
In general, systematic sampling is superior to
stratified random sampling when only one or
two samples per stratum are taken for
estimating the mean (Cochran, 1977) or
when there is a known pattern of
management measure implementation.
Gilbert (1987) reports that systematic
sampling is equivalent to simple random
sampling in estimating the mean if the target
population has no trends, strata, or
correlations among the population units.
Cochran (1977) notes that on the average,
simple random sampling and systematic
sampling have equal variances. However,
Cochran (1977) also states that for any single
population for which the number of sampling
units is small, the variance from systematic
sampling is erratic and might be smaller or
larger than the variance from simple random
sampling.
Gilbert (1987) cautions that any periodic
variation in the target population should be
known before establishing a systematic
sampling program. Sampling intervals equal
to or multiples of the target population's
cycle of variation might result in biased
estimates of the population mean.
Systematic sampling can be designed to
capitalize on a periodic structure if that
structure can be characterized sufficiently
(Cochran, 1977). A simple or stratified
random sample is recommended, however,
in cases where the periodic structure is not
well known or whether the randomly
selected starting point is likely to have an
impact on the results (Cochran, 1977).
Gilbert (1987) notes that assumptions about
the population are required in estimating
population variance from a single systematic
sample of a given size. There are, however,
systematic sampling approaches that do
support unbiased estimation of population
variance. They include multiple systematic
sampling, systematic stratified sampling, and
two-stage sampling (Gilbert, 1987). In
multiple systematic sampling, more than one
systematic sample is taken from the target
population. Systematic stratified sampling
involves the collection of two or more
systematic samples within each stratum.
2.1.3. Measurement and Sampling
Errors
In addition to making sure that samples are
representative of the sample population, it is
also necessary to consider the types of bias
or error that might be introduced into the
study. Measurement error is the deviation of
a measurement from the true value (e.g., the
percent compliance with SMA specifications
was estimated as 23 percent and the true
value was 26 percent). A consistent under-
or overestimation of the true value is
referred to as measurement bias. Random
-------
Chapter 2
• '-Ijsirvestjfe^o./'
1
* - < %' -'
3
4
5
6
s 7 ' '
8
9
10
11
12 ' • -
• • •
:> 142 * ' ~
143
144
145
146
, 147 " b"
148
149
150
,-.'* Witter Type
Stream
, " Stream.^ " 1
Lake/Pond
Stream
Lake/Pond
Lake/Pond
'/• Lake/Pond ,
Stream
Stream
Lake/Pond
Stream
Stream ,.
• • •
LakeVPond '
Lake/Pond
Stream
Lake/Pond
Stream
£ -Stream
Lake/Pond
Stream
Lake/Pond
^Ownership, '-'c
Industry
v Private Non-industrial
Industry
Industry
Private Non-industrial
Private Non-industrial
^/-' .Industry- - " ,
State
Federal
Private Non-industrial
Industry
<•*' „, - State
• • •
' " "industry
State
State
Industry
Private Non-industrial
, - RhWteiNolfeindisl'ial ,
Industry
Industry
Federal
S 0otttity.Co«iei/'
14
r 3^; -
12
11
7
4
3
10
10
5
3
- 6
• • •
' - ,- \\ * \
7
15
15
8
9 , ' .
1
12
11
Figure 2-4a. Systematic sampling from a listing of harvest sites. From a listing of all harvest
sites of interest, an initial site (Harvest Site No. 2) was chosen randomly from among the first ten
sites on the list. Every fifth site listed subsequently was then selected for sampling.
\
O
Figure 2-4b. Systematic sampling from a map. Dots
(• and O) represent harvest sites of interest. A
single point on the map (a) and one of the harvest
sites were randomly selected. A line was stretched
outward from the point to (and beyond) the selected
harvest site. The line was then rotated about the
map and every fifth dot that it touched was selected
for sampling (open dots—O). The direction of
rotation was determined prior to selection of the
point of the line's origin and the beginning harvest
site. The shaded lines on the map could Represent
county boundaries, soil type, watershed, oY some
other boundary, but were not used for the sampling.
process.
-------
Sampling Design
Chapter 2
sampling error arises from the variability
from one population unit to the next (Gilbert,
1987), explaining why the proportion of
operators using a certain'BMP differs from
one survey to another.
The goal of sampling is to obtain an accurate
estimate by reducing the sampling and
measurement errors to acceptable levels,
while explaining as much of the variability
as possible to improve the precision of the
estimates (Gaugush, 1987). Precision is a
measure of how close an agreement there is
between individual measurements of the
same population. The accuracy of a
measurement refers to how close the
measurement is to the true value. If a study
has low bias and high precision, the results
will have high accuracy. Figure 2-5
illustrates the relationship between bias,
pfecision, and accuracy.
As suggested earlier, numerous sources of
variability should be accounted for in
developing a sampling design. Sampling
errors are introduced by virtue of the natural
variability within any given population of
interest. Since sampling errors relate to MM
or BMP implementation, the most effective
method for reducing such errors is to
carefully determine the target population and
to stratify the target population to minimize
the nonuniformity in each stratum.
Measurement errors can be minimized by
ensuring that site inspections are well
designed. If data are collected by sending
staff out to inspect randomly selected harvest
sites, the approach for inspecting the harvest
sites should be consistent. For example,
how do field personnel determine the percent
of adequate SMAs, or what is the basis for
determining whether a BMP has been
properly implemented?
Reducing sampling errors below a certain
point (relative to measurement errors) does
not necessarily benefit the resulting analysis
because total error is a function of the two
types of errors. For example, if
measurement errors such as response or
interviewing errors are large, there is no
point in taking a huge sample to reduce the
sampling error of the estimate since the total
error will be primarily determined by the
measurement error. Measurement error is of
particular concern when landowner surveys
are used for implementation monitoring.
Likewise, reducing measurement errors
would not be worthwhile if only a small
sample size were available for analysis
because there would be a large sampling
error (and therefore a large total error)
regardless of the size of the measurement
error. A proper balance between sampling
and measurement errors should be
maintained because research accuracy limits
effective sample size and vice versa
(Blalock, 1979).
2.1.4. Estimation and Hypothesis
Testing
Rather than presenting every observation
collected, the data analyst usually
summarizes major characteristics with a few
descriptive statistics. Descriptive statistics
include any characteristic designed to
summarize an important feature of a data set.
A point estimate is a single number that
represents the descriptive statistic. Statistics
common to implementation monitoring
-------
Chapter 2
(a)
(b)
(c)
(d)
Figure 2-5. Graphical presentation of the relationship between bias, precision, and accuracy
(after Gilbert, 1987). (a): high bias + low precision = low accuracy; (b): low bias + low precision
= low accuracy; (c): high bias + high precision = low accuracy; and (d): low bias + high precision
= high accuracy.
include proportions, means, medians, totals,
and others. When estimating parameters of a
population, such as the proportion or mean,
it is useful to estimate the confidence
interval. The confidence interval indicates
the range in which the true value lies. For
example, if it is estimated that 65 percent of
waterbars on skid trails were installed in
accordance with design standards and
specifications and the 90 percent confidence
limit is ±5 percent, there is a 90 percent
chance that between 60 and 70 percent of the
waterbars were installed correctly.
Hypothesis testing should be used to
determine whether the level of MM and
BMP implementation has changed over time.
The null hypothesis (H0) is the root of
hypothesis testing. Traditionally, H0 is a
statement of no change, no effect, or no
difference; for example, "the proportion of
properly installed waterbars after operator
-------
Sampling Design
Chapter
training is equal to the proportion of
properly installed waterbars before operator
training." The alternative hypothesis (HJ is
counter to H0, traditionally being a statement
of change, effect, or difference, for example.
If H0 is rejected, Ha is accepted. Regardless
of the statistical test selected for analyzing
the data, the analyst must select the
significance level (a) of the test. That is, the
analyst must determine what error level is
acceptable based on the needs of decision
makers. There are two types of errors in
hypothesis testing:
Type I: H0 is rejected when H0 is really
true.
Type II: H0 is accepted when H0 is really
false.
Table 2-2 depicts these errors, with the
magnitude of Type I errors represented by a
and the magnitude of Type n errors
represented by p. The probability of making
a Type I error is equal to the a of the test
and is selected by the data analyst. In most
cases, managers or analysts will define 1-a
to be in the range of 0.90 to 0.99 (e.g., a
confidence level of 90 to 99 percent),
although there have been applications where
1-a has been set to as low as 0.80. Selecting
a 95 percent confidence level implies that the
analyst will reject the H0 when H0 is true
(i.e., a false positive) 5 percent of the time.
The same notion applies to the confidence
interval for point estimates described above:
a is set to 0.10, and there is a 10 percent
chance that the true percentage of properly
installed waterbars is outside the 60 to 70
percent range. This implies that if the
decisions to be made based on the analysis are
major (i.e., affect many people in adverse or
costly ways) the confidence level needs to be
greater. For less significant decisions (i.e.,
low cost ramifications) the confidence level
can be lower.
Type n error depends on the significance
level, sample size, and variability, and which
alternative hypothesis is true. Power (1-fi) is
defined as the probability of correctly
rejecting H0 when H0 is false. In general,
for a fixed sample size, a and p vary
inversely. For a fixed a, p can be reduced
by increasing the sample size (Remington
and Schorkrl970).
Table 2-2. Errors in hypothesis testing.
Decision
Accept H0
Reject H0
State of Affairs in the Population
\ H6 Is True
1-a
(Confidence level)
a
(Significance level)
(Type I error)
H0 (s False
3
(Type II error)
1-P
(Power)
-------
Chapter 2
2.2. SAMPLING CONSIDERATIONS
In a document of this brevity, it is not
possible to address all the issues that face
technical staff who are responsible for
developing and implementing studies to track
and evaluate the implementation of nonpoint
source control measures. For example,
when is the best time to implement a survey
or do on-site visits? In reality, it is difficult
to pinpoint a single time of the year. Some
BMPs can be checked any time of the year,
whereas others have a small window of
opportunity. If the goal of the study is to
determine the effectiveness of an operator
education program, sampling should be
timed to ensure that there was sufficient time
for outreach activities and for the operators
to implement the desired practices.
Furthermore, field personnel must have
approval to perform a site visit on each tract
of land to be sampled. Where access is
denied, a randomly selected replacement site
is needed.
2.2.1. Site Selection
From a study design perspective, all of these
issues must be considered together when
determining the sampling strategy. Site
selection criteria will differ from state to
state depending on the type of forestry
practiced in the state, physical landscape,
and intended purposes for the information
obtained from the implementation
monitoring. The following list indicates the
typical site selection criteria culled from
existing state implementation monitoring
programs. (The corresponding state postal
code is presented in parentheses.)
• Site size: minimum of 5 or 10 acres,
depending on the region of the state
(MN); minimum 10 acres (SC);
minimum 5 acres (MT); minimum 20
acres (ID).
• Proximity to a stream (perennial or
intermittent): within 300 feet of a
stream, or a lake of at least 10 acres
surface area (FL); within 200 feet of a
stream (MT); sites did not have to be
associated with streams or wetlands
(SC); within 150 feet of a class II
stream (ID).
• Time of harvest: within the past 1
year (SC); 1-3 years prior to the audit
(MT); within 2 years of harvest (FL).
• Site preparation: only sites that had
not been site prepared (SC); either
slash piled and burned or waiting
burning, or slash broadcast and
scheduled to be burned (MT).
• Volume harvested:
(MT).
at least 7 MBF/ac
• Compatibility with previous surveys:
sales had to meet the selection criteria
of a previous study for comparability
purposes (MT).
Other criteria that might be considered
include erosion risk (e.g., more sampling
sites could be placed in high-erosion-risk
areas than in low-risk erosion areas) and
beneficial use (bias sampling toward high use
and/or sensitive areas).
-------
Sampling Design
Chapter 2
2.2.2. Data to Support Site Selection
A list of harvest sites from, which to choose
those to be surveyed can be created from
information obtained from timber harvesters.
Depending on the state, the information is
often in the form of harvest plans and timber
sale contracts. These sources of information
normally include:
• U.S. Forest Service offices.
* The state forestry agency,
department, or division (for state
lands and nonindustrial private).
* Private timber companies (Ehinger
and Potts, 1991).
In addition, the Bureau of Land Management
(BLM) manages a significant acreage of
federal property and may have valuable
information (IDDHW, 1993). Aerial
photographs of the areas to be surveyed can
be used to identify recent harvest sites as
well, and this method of identifying the sites
tends to remove site selection biases due to
the distance of sites from roads or other
forms of inconvenience that otherwise might
make them less apt to be chosen .for a
survey.
The data necessary to select sites for BMP
tracking will naturally depend on the site
selection criteria. For instance, if sites must
meet a minimum of board feet harvested, it
will be necessary to know harvest volumes hi
order to select appropriate sites. The
amount of data needed will increase as the
number of site selection criteria increase,
and this should be taken into account when
deciding on the criteria, especially given the
possibility that some of the data or types of
data collected might be unavailable or
unreliable.
2.2.3. Example State and Federal
Programs
Several states and federal agencies have
implemented programs for developing their
implementation monitoring programs. This
section describes those implemented by
Florida, Montana, Idaho, and the USDA
Forest Service.
2.2.3.1. Florida
In Florida, the following site selection
criteria are used (Vowell and Gilpin, 1994):
• All ownership classes are included.
t
• Only the northernmost 37 counties are
included because most forestry
activities occur in these counties.
• Timber harvesting, site preparation,
tree planting, or some combination
must have occurred within the past 2
years and within 300 feet of an
intermittent or perennial stream or a
lake 10 acres or larger.
• Each county has a predetermined
number of survey sites based on the
level of timber removal reported by
the Forest Service.
• Sites are selected from fixed-wing
aircraft using a random, predeter-
mined flight pattern in each county.
-------
Chapter 2
County foresters randomly select
qualifying sites along the flight
pattern until they have located the
number of survey sites assigned to
their county.
This approach is a type of stratified random
sampling. The entire population (entire
state) is first divided into strata containing
the northernmost 37 counties based on prior
information that indicated most forestry
operations occur in those counties. These
strata are still too large to conduct random
sampling; therefore, the criteria described
above are used to reduce the strata to a
manageable number given available
resources.
2.2.3.2. Montana and Idaho
Montana is interested in certain types of
information related to BMP implementation,
so they stratify their sample before selecting
sites. They follow these steps:
• Information on the sites (e.g.,
ownership, erosion hazard) is
compiled by watershed or basin.
i
• A list of all sales and information on
them is compiled for each basin for
the time period of interest (usually
within 1-2 years of harvest date).
• Sites that do not meet the selection
criteria are eliminated.
• Sites that do meet the selection
criteria are ground-truthed.
This is a stratified random approach: Within
drainage basins, sites are stratified first by
ownership and then by erosion hazard
(Ehinger and Potts, 1991; Schultz, 1992).
Idaho also uses this approach, stratifying
sites by geographic region and administrative
category. This ensures that differences in
MM and BMP implementation among
different soil, geologic, and administrative
groupings are not lost as would be the case if
simple random sampling were used
(IDDHW, 1993).
2.2.3.3. U.S. Forest Service
The U.S. Forest Service (USDA, 1992) has
developed a monitoring system for Region 5
of the Forest Service, Best Management
Practice Evaluation Program (BMPEP), with
the following objectives:
• Assess the degree of implementation
ofBMPs.
• Determine which BMPs are effective.
• Determine which BMPs need
improvement or development.
• Fulfill Forest Land and Resource
Management Plan BMP monitoring
commitments.
• Provide a record of performance for
management of nonpoint source
pollution in Region 5 of the Forest
Service.
These objectives are met through three
evaluation phases: Administrative, on-site,
and in-channel. In general, the first two
-------
Sampling Design
Chapter 2
phases deal with issues related to
implementation monitoring, with
administrative evaluation primarily
addressing programmatic evaluation and on-
site evaluation dealing primarily with
individual practices. In-channel evaluation is
an example of effectiveness monitoring.
In the BMPEP, forests are assigned the
number and types of evaluations to be
completed each year. To support statistical
inference, the evaluations assigned to each
forest must be performed at randomly
identified sites. Sites to be evaluated are
identified in two ways: Randomly and by
selection ("selected" sites).
Randomly identified sites are essential for
making statistical inferences regarding the
implementation and effectiveness of BMPs.
Random sites are picked from a pool of sites
that meet specified criteria.
Selected sites are identified in various ways:
* Identified as part of a monitoring plan
prescribed in an environmental
assessment, environmental impact
study, or land management plan.
• Identified as part of a Settlement of
Negotiated Agreement.
• Part of a routine site visit.
* Follow-up evaluations upstream or In-
channel Evaluation Sites, to discover
sources of problems.
• Sites that are of particular interest to
site administrators, specialists, and/or
management due to their sensitivity,
uniqueness, and other factors.
• Selected for a particular reason
specific to local needs.
It is important to note that for statistical
inference, the sample pool can only contain
the randomly identified sites, not the
"selected" sites. Selected sites must be
clearly identified and kept separate from the
random sites during data storage and
analysis. Because on-site evaluation
addresses a range of practices, corresponding
methods are provided for developing sample
pools for randomly selected sites. For
example, the sample pool for SMAs should
be developed using a Sale Area Map from
the Pool of Timber Sales and counting the
number of units that have designated SMAs.
This constitutes the SMA sample pool.
The data obtained from the sources discussed
above may not be precisely what are
required by the state conducting the
implementation monitoring survey. Ehinger
and Potts (1991) report the following
difficulties encountered in using data from
the National Forest Service database:
• Flawed assumptions concerning the
age of a harvest (some were found to
be too old to meet the survey
criteria).
• Uncertain age of roads.
• Units within 200 feet of stream on
paper were in fact farther than 200
feet from a stream.
-------
Chapter 2
The following difficulties were associated
with private nonindustrial forest units:
• Inadequate database to identify sales
meeting the survey criteria.
• Permission to access landowner
property not granted.
Landowners who did grant permission were
interested primarily in demonstrating their
BMP efforts to the state forest department,
so this class of ownership was statistically
biased.
On private industrial forest sites, a backlog
of slash burnings on sale units was found,
preventing their use (because of survey
criteria), and state forests sites were mostly
found to ;be farther than 200 feet from a
stream, making them ineligible for the
survey. States must be aware that these
Icinds of limitations will be encountered.
2.3. SAMPLE SIZE CALCULATIONS
This section describes methods for
estimating sample sizes to compute point
estimates such as proportions and means, as
well as detecting changes with a given
significance level. Usually, several
assumptions regarding data distribution,
variability, and cost must be made to
determine the sample size. Some
assumptions might result hi sample size
estimates that are too high or too low.
Depending on the sampling cost and cost for
not sampling enough data, it must be decided
whether to make conservative or "best-
value" assumptions. Because the cost of
visiting any individual site or group of sites
is relatively constant, it is more economical
to collect a few extra samples rather than
realize later that additional data are needed.
In most cases, the analyst should probably
consider evaluating a range of assumptions
regarding the impact of sample size and
overall program cost. To maintain document
brevity, some terms and definitions used in
the remainder of this chapter are summarized
in Table 2-3. These terms are consistent
with those in most introductory-level
statistics texts, and more information can be
found there. Those with some statistical
training will note that some of these
definitions include an additional term
referred to as the finite population correction
term (1-fy), where (j> is equal to n/N. In
many applications, the number of population
units in the sample population (N) is large in
comparison to the number of population
units sampled (n), and (7-) can be ignored.
However, depending on the number of units
(harvest sites for example) in a particular
population, N can become quite small. N is
determined by the definition of the sample
population and the corresponding population
units. If (j> is greater than 0.1, the finite
population correction factor should not be
ignored (Cochran, 1977).
Applying any of the equations described in
this section is difficult when no historical
data set exists to quantify initial estimates of
proportions, standard deviations, means, or
coefficients of variation. To estimate these
parameters, Cochran (1977) recommends
four sources:
• Existing information on the same
population or a similar population.
-------
Sampling Design
Chapter 2
Table 2-3. Definitions used in sample size calculation equations.
N
= total number of population units in
sample population
•- number of samples
= preliminary estimate of sample
size
5 number of successes
= proportion of successes
= proportion of failures (1-p)
= im observation of a sample
- sample mean
= sample variance
- sample standard deviation
' total amount
= population mean
= population variance
= population standard deviation
Cv = coefficient of variation
s2(x) = variance of sample mean
4> = n/N (unless otherwise stated in
text)
s(x) = standard error (of sample mean)
1- = finite population correction factor
d = allowable error
d, = relative error
MX
P
o2
o
p = aln
q = l-p
s =
n-l
C = six
value corresponding to cumulative area of 1-
cc using the normal distribution (see Table
A1).
value corresponding to cumulative area of 1-
cc using the student t distribution with df
degrees of freedom (see Table A2).
* A two-step sample. Use the first-step
sampling results to estimate the
needed factors, for best design, of the
second step. Use data from both
steps to estimate the final precision of
the characteristic(s) sampled.
• A "pilot study" on a "convenient" or
"meaningful" subsample. Use the
results to estimate the needed factors.
Here the results of the pilot study
generally cannot be used in the
calculation of the final precision
because often the pilot sample is not
representative of the entire population
to be sampled.
• Informed judgment, or an educated
guess.
It is important to note that this document
only addresses estimating sample sizes with
traditional parametric procedures. The
methods described in this document should
-------
Chapter 2
be appropriate in most cases, considering the
type of data expected. If the data to be
sampled are skewed, as with much water
quality data, the analyst should plan to
transform the data to something symmetric,
if not normal, before computing sample sizes
(Helsel and Hirsch, 1995). Kupper and
Hafner (1989) also note that some of these
equations tend to underestimate the
necessary sample because power is not taken
into consideration. Again, EPA
recommends that if the analyst lacks a
background hi statistics, he/she should
consult with a trained statistician to be
certain that the approach, design, and
assumptions are appropriate to the task at
hand.
2.3.1. Simple Random Sampling
In simple random sampling, it is presumed
that the sample population is relatively
homogeneous and a difference in sampling
costs or variability is not expected. If the
cost or variability of any group within the
sample population were different, it might be
more appropriate to consider a stratified
random sampling approach.
To estimate the proportion of harvest sites
implementing a certain BMP or MM, such
that the allowable error, d, meets the study
precision requirements (i.e., the true
proportion lies betweenp-d andp+d with a
1-a confidence level), a preliminary estimate
of sample size can be computed as (Snedecor
and Cochran, 1980)
= 0.1
otherwise
(2-3)
where (J) is equal to n0/N. Table 2-4
demonstrates the impact on n of selecting/?,
a, d, dr, and N. For example, 278 random
-------
Sampling Design
Chapter 2
Table 2-4. Comparison of sample size as a function of p, a, d, dn and N for estimating
sroportions using equations 2-1 through 2-3.
Probability
of Success,
p
0.1
0.1
0.5
0.5
0.1
0.1
0.5
0.5
sianlP-
canc»
level, a
0.05
0.05
0.05
0.05
0.10
0.10
0.10
0.10
Allowable
errorv a
0.050
0.075
0.050
0.075
0.050
0.075
0.050
0.075
Restive •
«m», dr
0.500
0.750
0.100
0.150
0.500
0.750
0.100
0.150
Prrtftotrtisy
$ampi$
$teer«»
138
61
384
171
97
43
271
120
\ Sample Size, n
Number of Population Units In Sample
Population, W
500
108
55
217
127
82
43
176
97
750
117
61
254
139
86
43
199
104
1,000 :
121
61
278
146
97
43
213
107
2,000
138
61
322
171
97
43
238
120
targe N
138
61
384
171
97
43
271
120
samples are needed to estimate the
proportion of 1,000 harvest sites with
adequate SMAs to within ±5 percent
(<2=0.05) with a 95 percent confidence level,
assuming roughly one-half of harvest sites
have adequate SMAs.
Suppose the goal is to estimate the average
acreage per harvest site where erosion
controls are used. The number of random
samples required to achieve a desired margin
of error when estimating the mean (i.e., the
What sample size is necessary to estimate the
average number Qf acres per harvest site
using erosion controls to within ±2&ac»s?
What sample size is necessary to estimate the
average number of acres perharv$$t site
using erosion controls to within ±1QpepBBf>t?
true mean lies between ~x-d and Tt+d with a
1-a confidence level) is (Gilbert, 1987)
n =
(2-4)
If N is large, the above equation can be
simplified to
Since the Student's t value is a function of n,
Equations 2-4 and 2-5 are applied iteratively.
That is, guess at what n will be, look up
ti-a/2,n-i from Table A2, and compute a
revised n. If the initial guess of n and the
revised n are different, use the revised n as
the new guess, and repeat the process until
the computed value of n converges with the
guessed value. If the population standard
-------
Chapter 2
deviation is known (not too likely), rather
than estimated, the above equation can be
further simplified to
(2-6)
To keep the relative error of the mean
estimate below a certain level (i.e., the true
mean lies between ~%-dr H and ~X+dr X with a
1-a, confidence level), the sample size can be
computed with (Gilbert, 1987)
K =-
(2-7)
Cv is usually less variable from study to
study than are estimates of the standard
deviation, which are used in Equations 2-4
through 2-6. Professional judgment and
experience, typically based on previous
studies, are required to estimate Cv. Had Cv
been known, Z;.a/2 would have been used hi
place of ?;.a/2,n./ in Equation 2-7. If N is
large, Equation 2-7 simplifies to:
R = (ti-Ms-iCJdf (2-8)
For Company X, harvest sites range hi size
from 20 to 400 acres although most are less
than 80 acres in size. The goal of the
sampling program is to estimate the average
number of harvested acres using erosion
controls. However, the investigator is
concerned about skewing the mean estimate
with the few large sites. As a result, the
sample population for this analysis is the 430
harvested sites with less than 80 total acres.
The investigator also wants to keep the
relative error under 15 percent (i.e., dr —
0.15) with a 90 percent confidence level.
Unfortunately, this is the first study that
Company X has done and there is no
information about Cv or s. The investigator,
however, is familiar with a recent study done
by another company. Based on that study,
the investigator estimates the Cv as 0.6 and s
equal to 30. As a first-cut approximation,
Equation 2-6 was applied with Z;.a/2 equal to
1.645 and assuming N is large:
n =(1-645 *0.6/0.15)2 = 43.3 ~ 44 samples
Since n/N is greater than 0.1 and Cv is
estimated (i.e., not known), it is best to
reestimate n with Equation 2-7 using 44
samples as the initial guess of n. In this
case, ?7_B/2 n.j is obtained from Table A2 as
1.6811.
n =
(1.6811xQ.6/0.15)2
l+(l-6811x0.6/0.15)2/430
= 40.9 ~ 41 samples
Notice that the revised sample is somewhat
smaller than the initial guess of n. In this
case it is recommended to reapply the
Equation 2-7 using 41 samples as the revised
guess of n. In this case, //.0/2j/,../ is obtained
from Table A2 as 1.6839.
n =
(1.6839 xQ.6/0.15)2
1 +(1.6839 x0.6/0.15)2/430
= 41.0 ~ 41 samples
Since the revised sample size matches the
estimated sample size on which ^.a/2,n-j was
based, no further iterations are necessary
The proposed study should include 41
harvested sites randomly selected from the
430 sites with less than 80 total acres.
When interest is focused on whether the
level of BMP implementation has changed, it
is necessary to estimate the extent of
implementation at two different time periods.
-------
Sampling Design
Chapter 2
Alternatively, the proportion from two
different populations can be compared. In
either case, two independent random samples
are taken and a hypothesis test is used to
What sample size is necessary to determine
whether there is a 20 percent difference* in
BMP implementation before and after an
operator training program?
What sample size is necessary to detect a.
30-acre increase in average harvested
acreage per site using erosion controls when
comparing private and public timber sates?
determine whether there has been a
significant change in implementation. (See
Snedecor and Cochran (1980) for sample
size calculations for matched data.)
Consider an example in which the proportion
of waterbars that effectively divert water
from the skid trail will be estimated at two
time periods. What sample size is needed?
To compute sample sizes for comparing two
proportions, p-, audp2, it is necessary to
provide a best estimate for/?; andp2, as well
as specifying the significance level and
power (2-P). Recall that power is equal to
the probability of rejecting H0 when H0 is
false. Given this information, the analyst
substitutes these values into (Snedecor and
Cochran, 1980)
(2-9)
where Za and Z2p correspond to the normal
deviate. Although this equation assumes that
N is large, it is acceptable for practical use
(Snedecor and Cochran, 1980). Common
values of (ZB + Z2p)2 are summarized hi
Table 2-5. To account forpj aadpz being
estimated, Z should be replaced with t. In
lieu of an iterative calculation, Snedecor and
Cochran (1980) propose the following
approach: (1) compute n0 using Equation 2-
9; (2) round n0 up to the next highest
integer, /; and (3) multiply n0 by
(f+3)/(f+l) to derive the final estimate of «.
To detect a difference hi proportions of 0.20
with a two-sided test, a equal to 0.05, 7-P
equal to 0.90, and an estimate ofp2 and/?2
equal to 0.4 and 0.6, n0 is computed as
n ~ 10.51 K°-4X°-6) + (°-6)(°-4)] - 126.1
" * ' (0.6-0.4)2
Rounding 126.1 to the next highest integer,/
is equal to 127, and n is computed as 126.1
x 130/128 or 128.1. Therefore, 129 samples
hi each random sample, or 258 total
samples, are needed to detect a difference hi
proportions of 0.2. Beware of other sources
of information that give significantly lower
estimates of sample size. In some cases the
other sources do not specify 7-p; hi all
cases, it is important that an "apples-to-
apples" comparison is being made.
To compare the average from two random
samples to detect a change of 5 (i.e., %-#;),
the following equation is used:
(2-10)
52
Common values of (Za + Z2p)2 are
summarized hi Table 2-5. To account for Sj
and sz being estimated, Z should be replaced
with t. In lieu of an iterative calculation,
Snedecor and Cochran (1980) propose the
-------
Chapter 2
Table 2-5. Common values of (Za + Zzp)2 for estimating sample size for use with equations 2-9
and 2-10.
PQvfW,
1-p
0.80
0.85
0.90
0.95
0 99
a for Qne-skfed Test
£.01
10.04
11.31
13.02
15.77
21.65
0.05
6.18
7.19
8.56
10.82
15.77
0.10
4.51
5.37
6.57
8.56
13.02
a for Two-sided Test
0x01
11.68
13.05
14.88
17.81
24.03
O.OS
7.85
8.98
10.51
12.99
18.37
MO
6.18
7.19
8.56
10.82
15.77
following approach: (1) compute n0 using
Equation 2-10; (2) round n0 up to the next
highest integer,/; and (3) multiply n0 by
(f+3)/(f+l) to derive the final estimate of n.
Continuing the Company X example above,
where s was estimated as 30 acres, the
investigator will also want to compare the
average number of harvested acres that used
erosion controls to the average number of
harvested acres that used erosion controls hi
a few years. To demonstrate success, the
investigator believes that it will be necessary
to detect a 20-acre increase. Although the
standard deviation might change after the
operator training program, there is no
particular reason to propose a different s at
this point. To detect a difference of 20 acres
with a two-sided test, a equal to 0.05, 7-0
equal to 0.90, and an estimate of s} and s2
equal to 30, n0 is computed as
Rounding 47.3 to the next highest integer, /
is equal to 48, and n is computed as (47.3)»
(51/49) or 49.2. Therefore 50 samples hi
each random sample, or 100 total samples,
are needed to detect a difference of 20 acres.
2.3.2. Stratified Random Sampling
The key reason for selecting a stratified
random sampling strategy over simple
random sampling is to divide a
heterogeneous population into more
homogeneous groups. If populations are
grouped based on size (e.g., site size) when
there is a large number of small units and a
What sample size is necessary to estimate the
average SMA widtb per harvest site wben
th&re is a wtde variety of stream types and site
conditions?
n B 10.5l
202
(2-11)
few larger units, a large gain in precision
can be expected (Snedecor and Cochran,
1980). Stratifying also allows the
investigator to. efficiently allocate sampling
-------
Sampling Design
Chapter 2
resources based on cost. The stratum mean,
3CA, is computed using the standard approach
for estimating the mean. The overall mean,
"X,,, is computed as
IX**
A-l
(2-12)
where ch is the per unit sampling cost in the
h* stratum and nh is estimated as (Gilbert,
1987)
W, = K
(2-16)
where L is the number of strata and Wh is the
relative size of the hm stratum. Wh can be
computed as Nh/N where Nh and N are the
number of population units hi the hm stratum
and the total number of population units
across all strata, respectively. Assuming that
simple random sampling is used within each
stratum, the variance of Xw is estimated as
(Gilbert, 1987)
Nhnh
(2-13)
where nh is the number of samples in the h*
stratum and sh2 is computed as (Gilbert,
1987)
nh-l ,.,
(2-14)
There are several procedures for computing
sample sizes. The method described below
allocates samples based on stratum size,
variability, and unit sampling cost. If s2ftst)
is specified as V for a design goal, n can be
obtained from (Gilbert, 1987)
L
A»t
L
AM
il>"-'
(2-15)
In the discussion above, the goal is to
estimate an overall mean. To apply a
stratified random sampling approach to
estimating proportions, ph, pst, phqh, and
s2(pst) should be substituted for nh, xst, sh2,
and s2ftj in the above equations,
respectively.
To demonstrate the above approach, consider
the Company X example again. In addition
to the 430 sites that are less than 80 acres,
there are 100 sites that range hi size from 81
to 200 acres, 50 sites that range in size from
201 to 300 acres, and 20 sites that range in
size from 301 to 400 acres. Table 2-6
presents three basic scenarios for estimating
sample size. In the first scenario, sh and ch
are assumed equal among all strata. Using a
design goal of V equal to 100 and applying
Equation 2-15 yields a total sample size of
41.9 or 42. Since sh and ch are uniform,
these samples are allocated proportionally to
Wh, which is referred to as proportional
allocation. This allocation can be verified
by comparing the percent sample allocation
to WH. Due to rounding up, a total of 44
samples are allocated.
Under the second scenario, referred to as the
Neyman allocation, the variability between
strata changes, but unit sample cost is
constant. In this example, sh increases by 15
between strata.- Because of the increased
-------
Chapter 2
Table 2-6. Allocation of samples.
Harvest Site
Size (acres)
Number of
Harvest
Sites
M.)
Relative
Size
M
Standard
Deviation
($*)
Unit
Sample
Cost
(**>
Sample Allocation
Number
%
A} Proportional allocation {$„ and ah are constant)
20-80
81-200
201-300
301-400
430
100
50
20
0.7167
0.1667
0.0833
0.0333
30
30
30
30
1
1
1
1
31
7
4
2
70.5
15.9
9.1
4.5
Using Equation 2-15, n is equal to 41.9. Applying Equation 2-16 to each stratum yields a total of 44
samples after rounding up to the next integer.
8} Neyman allocation {ch is constant)
20-80
81-200
201-300
301-400
430
100
50
20
0.7167
0.1667
0.0833
0.0333
30
45
60
75
1
1
1
1
35
13
9
5
56.5
21.0
14.5
8.1
Using Equation 2-15, n is equal to 59.3. Applying Equation 2-16 to each stratum yields a total of 62
samples after rounding up to the next integer.
0) Allocation where sft and *» are not constant
20-80
81-200
201-300
301-400
430
100
50
20
0.7167
0.1667
0.0833
0.0333
30
45
60
75
1.00
1.25
1.50
2.00
38
12
8
4
61.3
19.4
12.9
6.5
Using Equation 2-15, n is equal to 60.0. Applying Equation 2-16 to each stratum yields a total of 62
samples after rounding up to the next integer.
variability in the last three strata, a total of
59.3 or 60 samples are needed to meet the
same design goal. So while more samples
are taken in every stratum, proportionally
fewer samples are needed in the smaller site
size group. For example, using proportional
allocation, more than 70 percent of the
samples are taken in the 20- to 80-acre site
-------
Sampling Design
Chapter 2
size stratum, whereas approximately 57
percent of the samples are taken hi the same
stratum using the Neyman allocation.
Finally, introducing sample cost variation
will also affect sample allocation. In the last
scenario it was assumed that it is twice as
expensive to evaluate a harvest site from the
largest size stratum than to evaluate a harvest
site from the smallest size stratum. In this
example, roughly the same total number of
samples are needed to meet the design goal,
yet more samples are taken hi the smaller
size stratum.
2.3.3. Cluster Sampling
Cluster sampling is commonly used when
there is a choice between the size of the
sampling unit (e.g., skid trail versus harvest
site). Li general, it is cheaper to sample
larger units than smaller units, but the results
tend to be less accurate (Snedecor and
Cochran, 1980). Thus, if there is not a unit
sampling cost advantage to cluster sampling,
it is probably better to use simple random
sampling. To decide whether to perform
cluster sampling, it will probably be
necessary to perform a special investigation
to quantify sampling errors and costs using
the two approaches.
Perhaps the best approach to explaining the
difference between simple random sampling
and cluster sampling is to consider an
example set of results. In this example, the
investigator did an evaluation to determine
whether harvest sites had adequate SMAs.
Since the state had timber harvesting
activities across the state, the investigator
elected to inspect 10 harvest sites along each
randomly selected river. Table 2-7 presents
the number of harvest sites along each river
that had the recommended BMPs. The
overall mean is 5.6; a little more than one-
half of the sites have implemented the
recommended BMPs. However, note that
since the population unit corresponds to the
10 sites collectively, there are only 30
samples and the standard error for the
proportion of sites using recommended
BMPs is 0.035. Had the investigator
incorrectly calculated the standard error
using the random sampling equations, he or
she would have computed 0.0287, nearly a
20 percent error.
Since the standard error from the cluster
sampling example is 0.035, it is possible to
estimate the corresponding simple random
sample size to obtain the same precision
using
Pq _ (0.56X0.44) _
s(p)2 0.0352
(2-17)
Is collecting 300 samples using a cluster
sampling approach cheaper than collecting
about 200 simple random samples? If so,
cluster sampling should be used; otherwise,
simple random sampling should be used.
2.3.4. Systematic Sampling
It might be necessary to obtain an estimate of
the proportion of harvest sites where cable
yarding was implemented using site
inspections. Assuming a record of harvest
sites (where cable yarding was specified in
the timber sale contract or administration
file) is available in a sequence unrelated to
the manner in which this BMP would be
implemented (e.g., in alphabetical order by
-------
Chapter 2
Table 2-7. Number of harvest sites (out of 10) implementing recommended BMPs.
3 95 76 4
5 77475
8 47453
6
3
3
3 5
8 4
Q Q.
5
6
7
Grand Total = 168
x = 5.6 p = 5.6/1 0=0.560
5=1.923 5=1.923/10=0.1923
Standard error using cluster sampling: s(p)=0.1923/(30)05=0.035
Standard error if simple random sampling assumption had been incorrectly
s(p;=((0.56)(1-0.56)/300)05 =0.0287
used:
the operator's name), a systematic sample
can be obtained by selecting a random
number r between 1 and «, where n is the
number required in the sample (Casley and
Lury, 1982). The sampling units are then r,
r + (N/n), r + (2N/n), ..., r + (n-l)(N/n),
where N is total number of available records.
If the population units are hi random order
(e.g., no trends, no natural strata,
uncorrelated), systematic sampling is, on
average, equivalent to simple random
sampling.
Once the sampling units (hi this case,
specific harvest sites) have been selected,
site inspections can be made to assess the
extent of compliance with cable yarding
standards and specifications.
-------
-------
CHAPTERS. METHODS FOR EVALUATING DATA
3.1. INTRODUCTION
Once data have been collected, it is
necessary to statistically summarize and
analyze the data. EPA recommends that the
data analysis methods be selected before
collecting the first sample. Many statistical
methods have been computerized in easy-to-
use software that is available for use on
personal computers. Inclusion or exclusion
in this section does not imply an
endorsement or lack thereof by the U.S.
Environmental Protection Agency.
Commercial-off-the-shelf software that
covers a wide range of statistical and
graphical support includes SAS, Statistica,
Statgraphics, Systat, Data Desk (Macintosh
only), BMDP, and IMP. Numerous
spreadsheets, database management
packages, and other graphics software can
also be used to perform many of the needed
analyses. In addition, the following
programs, written specifically for
environmental analyses, are also available:
SCOUT: A Data Analysis Program,
EPA, NTIS Order Number PB93-
505303.
WQHYDRO (WATER
QUALITY/HYDROLOGY
GRAPHICS/ANALYSIS SYSTEM),
Eric R. Aroner, Environmental
Engineer, P.O. Box 18149, Portland,
OR 97218.
WQSTAT, Jim C. Loftis, Department of
Chemical and Bioresource Engineering,
Colorado State University, Fort Collins,
CO 80524.
Computing the proportion of sites
implementing a certain BMP or the average
number of acres that are under a certain
BMP follows directly from the equations
presented in Section 2.3 and is not repeated.
The remainder of this section is focused on
evaluating changes in BMP implementation.
The methods provided in this section provide
only a cursory overview of the type of
analyses that might be of interest. For a
more thorough discussion on these methods,
the reader is referred to Gilbert (1987),
Snedecor and Cochran (1980), and Helsel
and Hirsch (1995). Typically the data
collected for evaluating changes will
typically come as two or more sets of
random samples. In this case, the analyst
will test for a shift or step change.
Depending on the objective, it is appropriate
to select a one- or two-sided test. For
example, if the analyst knows that BMP
implementation will only go up as a result of
an operator education program, a one-sided
test could be formulated. Alternatively, if
the analyst does not know whether
implementation will go up or down, a two-
sided test is necessary. To simply compare
two random samples to decide whether they
are significantly different, a two-sided test is
used. Typical null hypotheses (H0) and
alternative hypotheses (HJ for one- and two-
sided tests are provided below:
One-sided test
H0: BMP Implementation (Post education)
<. BMP Implementation (Pre education)
-------
Methods for Evaluating Data
Chapter 3
Ha: BMP Implementation (Post education)
> BMP Implementation (Pre education)
Two-sided test
H0: BMP Implementation (Post education)
= BMP Implementation (Pre education)
Ha: BMP Implementation (Post education)
# BMP Implementation (Pre education)
Selecting a one-sided test instead of a two-
sided test results in an increased power for
the same significance level (Winer, 1971).
That is, if the conditions are appropriate, a
corresponding one-sided test is more
desirable than a two-sided test given the
same a and sample size. The manager and
analyst should take great care in choosing
one- or two-sided tests.
3.2. COMPARING THE MEANS FROM Two
INDEPENDENT RANDOM SAMPLES
The Student's t test for two samples and the
Mann-Whitney test are the most appropriate
tests for these types of data. Assuming the
data meet the assumptions of the t test, the
two-sample t statistic with n}+n2-2 degrees
of freedom is (Remington and Schork, 1970)
O, -*2) - A0
(3-1)
where n; and nz are the sample sizes of the
first and second data sets, respectively, and
r/ and ~X2 are the estimated means from the
first and second data sets, respectively. The
pooled standard deviation, sp, is defined by
Tests for Two Independent Random Samples
Test' Key Assumptions
Two-sample t
Both data sets must be
normally distributed ;
Data sets should have
equal variances* ,
Mann-Whitney • None
The standard forms of these tests require
independent random samples.
The variance homogeneity assumption can
be relaxed.
(3*2)
where Sj2 and s22 correspond to the estimated
variances of the first and second data sets,
respectively. The difference quantity (A0)
can be any value, but here it is set to zero.
A0 can be set to a non-zero value to test
whether the difference between the two data
sets is greater than a selected value. If the
variances are not equal, Snedecor and
Cochran (1980) can be used as a source for
methods for computing the t statistic. In a
two-sided test, the value from Equation 2-18
is compared to the t value from Table A2
with a/2 and n1+n2-2 degrees of freedom.
The Mann-Whitney test can also be used to
compare two independent random samples.
This test is very flexible since there are no
assumptions about the distribution of either
sample or whether the distributions have to
be the same (Helsel and Hirsch, 1995).
Wilcoxon (1945) first introduced this test for
equal-sized samples. Mann and Whitney
(1947) modified the original Wilcoxon's test
to apply it to different sample sizes. Here, it
-------
Chapter 3
is determined whether one data set tends to
have larger observations than the other.
If the distributions of the two samples are
similar except for location (i.e., similar
spread and skew), Ha can be refined to imply
that the median concentration from one
sample is "greater than," "less than," or
"not equal to" the median concentration
from the second sample. To achieve this
greater detail hi Ha, transformations such as
logs can be used.
Tables of Mann-Whitney test statistics (e.g.,
Conover, 1980) can be consulted to
determine whether to reject H0 for small
sample sizes. If rij and n2 are greater than or
equal to 10 observations, the test statistic can
be computed from the following equation
(Conoyer, 1980):
T - n
n+l
4(11 -L)
(3-3)
where
tij = number of observations in sample
with fewer observations,
n2 = number of observations in sample
with more observations,
n = n1 + n2,
T = sum of ranks for sample with fewer
observations, and
Rt = rank for the rth ordered observatipn
used in both samples.
Tj is normally distributed and Table Al can
be used to determine the appropriate
quantile. Helsel and Hirsch (1995) and
USEPA (1996) provide detailed examples for
both of these tests.
3.3. COMPARING THE PROPORTIONS FROM
Two INDEPENDENT SAMPLES
Consider the example in which the
proportion of waterbars that effectively
divert water from the skid trail has been
estimated during two time periods to be pj
andp2 using sample sizes of n1 and n2,
respectively. Assuming a normal
approximation is valid, the test statistic
under a null hypothesis of equivalent
proportions (no change) is
P \-P-L
\
(3-4)
where p is a pooled estimate of proportion
and is equal to fa+xj/frij+n^ and x1 and x2
are the number of successes during the two
time periods. An estimator for the
difference in proportions is simply prp2.
In an earlier example, it was determined that
129 observations hi each sample were needed
to detect a difference in proportions of 0.20
with a two-sided test, a equal to 0.05, 1-$
equal to 0.90. Assuming that 130 samples
were taken and p1 and p2 were estimated
from the data as 0.6 and 0.4, the test statistic
would be estimated as
0.6-0.4
= 3.22
(3-5)
0.5(0.5)| -- + -
130 130J
-------
Methods for Evaluating Data
! Chapter 3
Comparing this value to the t value from
Table A2 (a/2 = 0.025, df=258) of 1.96,
H0 is rejected.
3.4. COMPARING MORE THAN Two
INDEPENDENT RANDOM SAMPLES
The analysis of variance (ANOVA) and
Kruskal-Wallis are extensions of the
two-sample t and Mann-Whitney tests,
respectively, and can be used for analyzing
more than two independent random samples
when the data are continuous (e.g., average
SMA width). Unlike the t test described
earlier, the ANOVA can have more than one
factor or explanatory variable. The Kruskal-
Wallis test accommodates only one factor,
whereas the Friedman test can be used for
two factors. In addition to applying one of
the above tests to determine if one of the
samples is significantly different from the
others, it is also necessary to perform
postevaluations to determine which of the
samples is different. This section
recommends Tukey's method to analyze the
raw or rank-transformed data only if one of
the previous tests (ANOVA, rank-
transformed ANOVA, Kruskal-Wallis,
Friedman) indicates a significant difference
between groups. Tukey's method can be
used for equal or unequal sample sizes
(Helsel and Hirsch, 1995). The reader is
cautioned, when performing an ANOVA
using standard software, to be sure that the
ANOVA test used matches the data. USEPA
(1996) provides a more detailed discussion
on comparing more than two independent
random samples.
3.5. COMPARING CATEGORICAL DATA
In comparing categorical data it is important
to distinguish between whether the categories
are nominal (e.g., land ownership, county
location, type of BMP) or ordinal (e.g.,
BMP implementation rankings, low-medium-
high scales).
The starting point for all evaluations is the
development of a contingency table. In
Table 3-1, the preference of three BMPs is
compared to harvest site type in a
contingency table. In this case both
categorical variables are nominal. In this
example, 45 of the 102 observations on
federal lands used EMP^ There were a total
of 174 observations.
To test for independence, the sum of the
squared differences between the expected
(E^ and observed (Oy) count summed over
all cells is computed as (Helsel and Hirsch,
1995)
EE-
<=i j=\
(3-6)
where Ey is equal to A^/N. %ct is compared
to the 1-a quantile of the %2 distribution with
(m-l)(k-l) degrees of freedom (see Table
A3).
In the example presented in Table 3-1, the
symbols listed in the parentheses correspond
to the above equation. Note that k
corresponds to the three types of BMPs and
m corresponds to the three different types of
harvest site. Table 3-2 shows computed
values of EtJ and (Oij-
for the example data.
/Ey in parentheses
is equal to 14.60.
-------
Chapter 3
lafffiW^PP^
Table 3-1. Contingency table of harvest site type and implemented BMP.
Harvest Site Type
Private
Federal
State
Column Total,^,-
BWH*, BNIPjr BWJPa
10(0,,)
45 (021)
8 (081)
63 (CO
30 (012)
32 (0,2)
3 (032)
65 (C2)
17(013)
25 (023)
4(033)
46 (C3)
Row Total,
Ar
57 (A,)
102(A2)
15(A3)
174(N)
Key to Symbols:
Oj = number of observations for the fth harvest site and yth BMP type
A, = row total for the Ah harvest site type (total number of observations for a given harvest site type)
Cj = column total for the /th BMP type (total number of observations for a given BMP type)
N = total number of observations
From Table A3, the 0.95 quantile of the %2
distribution with 4 degrees of freedom is
9.488. H0 is rejected; the selection of BMP
is not random among the different harvest
site types. The largest values in the
parentheses hi Table 3-2 give an idea as to
which combinations of harvest site type and
BMP are noteworthy. In this example, it
appears that BMP2 is preferred to BMPt hi
comparison to federal and state harvest sites.
Now consider that hi addition to evaluating
information regarding the harvest site and
BMP type, we also recorded a value from 1
to 5 indicating how well the BMP was
installed and maintained, with 5 indicating
the best results. In this case, the BMP
implementation rating is ordinal. Using the
same notation as before, the average rank of
observations hi row x, Rx, is equal to (Helsel
and Hirsch, 1995)
(3-7)
where At corresponds to the row total. The
average rank of observations in column j, D
is equal to
/=!
(3-8)
where Cj corresponds to the column total.
The Kruskal-Wallis test statistic is then
computed as
K = (N-}
Y,<
/=i
lf-N
N+l
N
JV-t-1 L
N
(3-9)
where K is compared to the x2 distribution
with k-1 degrees of freedom. This is the
most general form of the Kruskal-Wallis test
since it is a comparison of distribution shifts
rather than shifts hi the median (Helsel and
Hirsch, 1995).
-------
Methods for Evaluating Data
Table 3-2. Contingency table of expected harvest site type and implemented BMP. (Values
in parentheses correspond to (On-E9)2/Et.)
Harvest Site Type
Private
Federal
State
Column Total
B1WP,
20.64
(5.48)
36.93
(1.76)
5.43
(1.22)
63
BMP2
21.29
(3.56)
38.10
(0.98)
5.60
(1.21)
65
8NIP3
15.07
(0.25)
26.97
(0.14)
3.97
(0.00)
46
Row Total
57
102
15
174
Table 3-3 is a continuation of the previous
example indicating the BMP implementation
rating for each BMP type. For example, 29
of the 70 observations that were given a
rating of 4 are associated with BMP2. The
terms inside the parentheses of Table 3-3
correspond to the terms used hi Equations 3-
7 to 3-9, Note that k corresponds to the
three types of BMPs and m corresponds to
the five different levels of BMP
implementation. Using Equation 3-9 for the
data in Table 3-3, Kis equal to 14.86.
Comparing this value to 5.991 obtained from
Table A3, there is a significant difference in
the quality of implementation between the
three BMPs.
The last type of categorical data evaluation
considered in this chapter is that in which
both variables are ordinal. The Kendall rb
for tied data can be used for this analysis.
The statistic tb is calculated as (Helsel and
Hirsch, 1995)
(3-10)
where S, SSa, and SSC are computed as
allxy
(3-12)
(3-13)
To determine whether T6 is significant, S is
modified to a normal statistic, using
S-l
S+l
ifS>0
ifS<0
(3-14)
-------
Chapter 3
Table 3-3. Contingency table of implemented BMP and rating of installation and
maintenance.
BJWP
Implementation
Rating
1
2
3
4
5
Column Total, C,
BMP,
1 (0,,)
7 (021)
15(03,)
32 (041)
8 (051)
63 (C,)
BMP2
2(012)
3(0^)
16 (032)
29 (042)
15(052)
65 (C2)
BMP3
2(013)
5 (023)
26 (033)
9 (043)
4 (053)
46 (C3)
Row Total,
A
5(A1)
15 (A2)
57 (A3)
70 (A<)
27 (A5)
174(N)
Key to Symbols:
On = number of observations for the fth BMP implementation rating andyth BMP type
A, = row total for the fth BMP implementation rating (total number of observations for a given BMP implementation rating)
C, = column total for theyth BMP type (total number of observations for a given BMP type)
N = total number of observations
where
(3-15)
to (2509-1)7679.75 or 3.69. Comparing this
value to a value of 1.96, obtained from
Table Al (a/2=0.025), indicates that BMP
implementation is improving with time.
where Zs is zero if S is zero. The values of
at and ct are compute as Ai /N and Ct/N,
respectively.
Table 3-4 presents the BMP implementation
ratings that were taken in three separate
years. For example, 15 of the 57
observations that were given a rating of 3 are
associated with Year 2. Using Equations 3-
11 and 3-15, S and as are equal to 2,509 and
679.75, respectively. Therefore, Zs is equal
-------
Methods for Evaluaiirm Data
Chapter 3
Table 3-4. Contingency table of implemented BMP and sample year.
BMP
Implementation
Rating
1
2
3
4
5
Column Total, C}
c,
Yean, Year 2 Y*w3
2 (0,,) 1 (012) 2 (013)
5 (021) 7 (0^) 3 (023)
26 (031) 15 (032) 16 (033)
9 (041) 32 (042) 29 (043)
4(0S1) 8(052) 15(0S3).
46 (C,) 63 (C2) 65 (C3)
0.264 0.362 0.374
Row Total,
A,
5 (A,)
15 (A2)
57 (A3)
70 (A<)
27 (As)
174(N)
«,
0.029
0.086
0.328
0.402
0.155
Key to Symbols:
number of observations for the Ah BMP implementation rating and/th year
row total for the Ah BMP implementation rating (total number of observations for a given harvest type)
column total for the .Ah BMP type (total number of observations for a given year)
total number of observations
a,
-------
CHAPTER 4. CONDUCTING THE EVALUATION
4.1 INTRODUCTION
This chapter addresses the process of
determining whether forestry MMs or BMPs
are being implemented and whether they are
being implemented according to approved
standards or specifications. Guidance is
provided on what should be measured to
assess MM and BMP implementation, as
well as methods for collecting the
information, including physical site
evaluations, mail- and/or telephone-based
surveys, personal interviews, and aerial
reconnaissance and photography. Designing
survey instruments to avoid error and rating
MM and BMP implementation are also
discussed.
Evaluation methods are separated into two
types: Expert evaluations and self-
evaluations. Expert evaluations are those in
which actual field investigations are
conducted by trained personnel to gather
information on MM or BMP
implementation. Self-evaluations are those
in which answers to a predesigned
questionnaire or survey are provided by
harvesters and/or landowners associated with
the survey site. Self-evaluations might also
include examination of materials related to a
harvest, such as harvest plans and records of
violations of forestry regulations. Extreme
caution should be exercised when using data
from self-evaluations as the basis for
assessing MM or BMP implementation since
they are not typically reliable for this
purpose. Each of these evaluation methods
has advantages and disadvantages that should
be considered before to deciding which one
to use or in what combination to use them.
Aerial reconnaissance and photography can
be used to support either evaluation method.
Self-evaluations are useful for collecting
information on landowner or harvester
awareness of MMs or BMPs, dates of
harvest, harvest site conditions, which MMs
or BMPs were implemented, and whether the
assistance of a professional forester was
used. However, the type of or level of detail
of information that can be obtained from
self-evaluations might be inadequate to
satisfy the objectives of a MM/BMP
compliance survey. If this is the case, expert
evaluations might be called for. Expert
evaluations are necessary if information on
MM/BMP implementation that is more
detailed or more reliable than that that can be
obtained with self-evaluations is required,
such as an objective assessment of the
adequacy of MM/BMP implementation, the
degree to which site-specific factors (e.g.,
slope, soil type, or presence of a water body)
influenced MM/BMP implementation, or the
need for changes in standards and .
specifications for MM/BMP implementation.
Sections 4.3 and 4.4 discuss expert
evaluations and self-evaluations,
respectively, in more detail.
Expert evaluations of implementation of
forestry MMs or BMPs generally occur after
the harvest has occurred (see Example), and
direct observation of the adequacy of
implementation of many BMPs (e.g.,
preharvest planning, pesticide applications,
or construction of temporary roads) might
-------
Conductinsz the Evaluation
Chapter 4
not be possible. However, evidence of
proper BMP implementation is often present
at harvest sites. For instance, evidence of
Example... Timing of site evaluations.
U.S. Forest Service, Southwest Region: After
fogging to within approximately one year after
the site is harvested (USDA, 1992).
South Carolina: One year or less than after
the site is harvested (Adams, 1994).
Florida: Two years or tess after the site is
harvested (Vowell and Gilpin, 1994).
excessive skidding in SMAs, vegetation kills
due to pesticide use in SMAs, and poorly
restored stream banks and stream beds where
temporary stream crossings were located is
often observable during site evaluations.
Supplemental information on aspects of
harvest operations that cannot be observed
directly during site evaluations might also be
obtained from self-evaluations.
Aerial reconnaissance and photography is
another means available for collecting
information on harvests, though many of the
MMs/BMPs employed for forestry are
difficult if not impossible to identify on
aerial photographs. For this reason, aerial
reconnaissance and photography are most
useful for identifying potential survey sites
and monitoring harvest site regeneration,
forest conditions, and some water quality
conditions (e.g., sediment runoff, algal
blooms). Aerial reconnaissance and
photography are discussed hi more detail hi
Section 4.5.
The general types of information obtainable
with self-evaluations are listed in Table 4-1.
Regardless of the approaches) used, proper
and thorough preparation for the evaluation
is the key to success.
4.2 CHOICE OF VARIABLES
Once the objectives of a BMP
implementation or compliance survey have
been clearly defined, the most important
factor in the assessment of MM or BMP
implementation is the determination of which
variable(s) to measure. A good variable
provides a direct measure of how well a
BMP was implemented. Individual variables
should provide measures of different factors
related to BMP implementation. The best
variables are those which are measures of the
adequacy of MM or BMP implementation
and are based on quantifiable expressions of
conformance with state standards and
specifications. As the variables used become
less directly related to actual MM or BMP
implementation, their accuracy as measures
of BMP implementation decreases.
Examples of useful variables include width
of streamside management areas, slope of
landing areas, and size of culverts, all of
which would be expressed hi terms of
conformance with applicable state standards
and specifications. Less useful variables
measure factors that are related to BMP
implementation but do not necessarily
provide an accurate measure of their
implementation. Examples of such variables
include the number of miles of forest roads
constructed and the number of preharvest
plans submitted to the state forestry agency,
department, or division (hereafter referred to
as the state forestry authority). Although
these variables relate to MM and BMP
-------
Conductinu the
Table 4-1. General types of information obtainable with self-evaluations and expert
evaluations.
Information Obtainable from Self-Evaluations
Harvest applications and management plans associated with the harvest (e.g., preharvest,
road, fire, forest chemical) might be available for review prior to site evaluations and can
provide much background information, such as:
Type of ownership (private industrial, private nonindustrial, federal, other public)
Total acreage under management
Acres/board feet harvested
Surface water body types on harvest site
Soil type
Ecological characterization of harvested area (e.g., habitat type, significant wildlife)
Presence of critical wildlife habitat
Species harvested
Harvest/management history of the harvested site
Use of cable yarding or ground skidding
Chemicals (e.g., pesticides, fertilizers) applied
Dates of plan preparation and revisions
Locations of roads and road structures, SMAs, loading areas, etc.
MMs and BMPs applied during harvest
Map
Conversations with harvesters and landowners can be used to verify information obtained
from applications or can yield supplemental information, such as:
• Dates of harvest
• Ambient conditions during applications
• Variations from harvest plan
• Problems encountered during harvest
• Types of equipment used during harvest and in SMAs
• Timing, location, and rate of chemical applications
Information Requiring Expert Evaluations
Expert evaluations are necessary to verify information obtained from self-evaluations and
records and to assess the actual adequacy of MM and BMP implementation. Expert
evaluations are necessary to:
• Assess design adequacy
• Assess installation adequacy
• Assess the appropriateness of operation methods and overall management
• Confirm information obtained from self-evaluations %
-------
Conducting the Evaluation
Chapter 4
implementation, they provide no real
information on whether the MMs and BMPs
are actually being implemented or whether
they are being implemented properly.
Variables generally will not directly relate to
MM implementation since most forestry
MMs are combinations of several BMPs.
Measures of MM implementation, therefore,
usually will be based on separate assessments
of two or more BMPs, and the
implementation of each BMP will be based
on a unique set of variables. Some examples
of BMPs related to EPA's Road
Construction and Reconstruction
Management Measure, variables for
assessing compliance with the BMPs, and
related standards and specifications that
might be required by state forestry
authorities are presented hi Figure 4.1.
Because harvesters choose to implement or
not implement MMs/BMPs based on site-
specific conditions, it is also appropriate to
apply varying weights to the variables
chosen to assess MM/BMP implementation
to correspond to site-specific conditions.
For example, variables related to slope
factors might be de-emphasized—and other,
more applicable variables emphasized
more—on relatively flat harvest sites.
Similarly, on a site with a water body,
variables related to SMAs, sediment runoff,
and chemical deposition (pesticide use,
fertilizer use) might be emphasized over
other variables to arrive at a site-specific
rating of the adequacy of MM/BMP
implementation.
The purpose for which the information
collected during an MM or BMP
implementation survey will be used is
another important consideration when
selecting variables. An implementation
survey can serve many purposes beyond the
primary purpose of assessing MM and BMP
implementation. For instance, for its 1993
BMP compliance survey, the South Carolina
Forestry Commission selected variables that
enabled it to assess compliance with each of
five categories of BMPs and overall
compliance with BMPs. In addition, the
Commission analyzed the effect of each of
16 additional variables on BMP compliance
(see Example). The purpose of the survey
was not only to assess BMP implementation,
but also to assess the relationship of various
conditions to the level of BMP compliance
(Adams, 1994).
Table 4-2 provides examples of useful and
less useful variables for the assessment of
implementation of the forestry MMs
developed by EPA (USEPA, 1993a). The
variables listed in the table are only
examples, and local or regional conditions
ultimately dictate which variables should be
used.
4.3 EXPERT EVALUATIONS
4.3.1 Site Evaluations
Expert evaluations are the best way to collect
reliable information on MM and BMP
implementation. They involve a person or
team of people visiting individual harvest
sites and speaking with harvest operators
and/or landowners to obtain information on
MM and BMP implementation. For many
MMs, assessing and verifying compliance
requires a site visit and evaluation. The
-------
Chapter 4
Conducting
Management Measure for Road Construction and Reconstruction
(1) Follow preharvest planning (as described in the Preharvest Planning Management Measure) when
constructing or reconstructing the roadway.
(2) Follow designs planned under the Preharvest Planning Management Measure for road surfacing and
shaping.
(3) Install road drainage structures according to designs planned under the Preharvest Planning
Management Measure and regional storm return period and installation specifications. Match these
drainage structures with terrain features and with road surface and prism designs.
(4) Guard against the production of sediment when installing stream crossings.
(5) Protect surface waters from slash and debris material from roadway clearing.
(6) Use straw bales, silt fences, mulching, or other favorable practices on disturbed soils on unstable cuts, fills,
etc.
(7) Avoid constructing new roads in SMAs to the extent practicable.
Related BMPs, measurement variables, and standards and specifications:
Management Measure
Practice
Preplan skid trail and landing
locations on stable soils and
avoid steep gradients and areas
that are landslide-prone or
erosion-prone, or have poor
drainage.
In moderately sloping terrain,
plan for road grades of less
than 10%, with an optimal
grade between 3% and 5%.
Vary road grades frequently to
reduce culvert and road
drainage ditch flows, road
surface erosion, and
concentrated culvert
discharges.
Design roads and skid trails to
follow the natural topography
and contour, minimizing
alteration of natural features.
1 Design cut-and-fill slopes to be
at stable angles, or less than the
normal angle of repose, to
minimize erosion and slope
failure potential.
Potential Measurement
Variables
Soil type or stability along skid
trails and at landings.
Gradients along skid trails and
at landings.
Categorization of terrain as flat,
moderate, steep.
Road grade estimated over
sections of road.
Steep terrain: Average distance
between changes in grade.
Natural slope of surrounding
terrain.
Slope of skid trails and at
landings.
Angle of cut-and-fill slopes.
Stability of soil type where cut-
and-fill slopes have been
installed.
Example Related Standards and
Specifications
• Minimum soil stability for skid
trails and landings.
• Maximum slope for skid trails
and landings.
Maximum road grade for a
given terrain slope.
Maximum distance between
changes in grade on steep
terrain.
Minimum/maximum distance
between drainage features for a
given terrain slope.
Maximum slope of skid trails
for a given terrain slope.
Maximum angle for cut-and-fill
slopes.
Figure 4-1. Potential variables and examples of implementation standards and specifications
that might be useful for evaluating compliance with the Road Construction and Reconstruction
Management Measure.
-------
Conduciinu the Hvalualion
Chapter 4
Table 4-2. Examples of variables related to management measure implementation.
Management
Measure
Prehan/est
Planning
Streamside
Management
Areas
Road
Construction/
Reconstruction
Road
Management
Timber
Harvesting
Site Preparation
and Forest
Regeneration
Fire
Management
Regeneration of
Disturbed Areas
Forest Chemical
Management
Wetlands Forest
-------
Chapter 4
Example... Variables used by the South
Carolina Forestry Commission during the
1993 BMP compliance survey. (Source:
Adams, 1994)
Qverati BMP compliance and compliance with
each of five categories of BMPs were
assessed:
* Road systems
* Road stream crossings
»Streamside management zones
• Log decks
* Harvesting operations
and for overall BMP compliance,
Sixteen additional variables were analyzed to
determine their relationship to BMP
compliance: •
«Presence of perennial streams
* Terrain type
»Percent slope
• Use of a professional forester
* Required compliance with BMPs
* Physiographic region
* Use of a sales contract
• Percent of site impacted
* Landowner category
* Familiarity of landowner with BMPs
* Logged under wet soft conditions
• Rutting severity
»Road construction applicability
* Soil drainage class
* Presence ofjurisdtetionat wetlands
* Harvest see
following should be considered before expert
evaluations are conducted:
• Obtaining permission from the
landowner. Without proper
authorization to visit a site
from a landowner, the
relationship between
landowners and the state
forestry authority, and any
future regulatory or
compliance action could be
jeopardized.
The type(s) of expertise needed
to assess proper
implementation. For some
MMs, a team of trained
personnel might be required at
a site evaluation to determine
whether MMs have been
implemented properly.
The activities that should
occur during a site evaluation.
This information is necessary
for proper and complete
preparation for the site visit,
so that the evaluation can be
completed hi a single visit and
at the proper tune.
The method of rating the
MMs/BMPs. MM and BMP
rating systems are discussed
below.
Consistency among evaluation
teams and among site
evaluations. Proper training
and preparation of site
evaluation team members are
crucial to ensure accuracy and
consistency.
The collection of information
while at a site. Information
collection should be facilitated
with preparation of data
-------
Conducting the Evaluation
Chapter 4
collection forms that include
any necessary MM and BMP
rating information needed by
the evaluation team members.
• The content and format of
postevaluation discussions.
Site evaluation team members
should bear in mind the value
of postevaluation discussion
among team members. Notes
can be taken during the
evaluation concerning any
items that would benefit from
group discussion.
Evaluators might consist of a single person
suitably trained in silvicultural site
evaluation to a group of professionals with
various expertise. The composition of an
evaluation team will depend on the types of
MMs or BMPs being evaluated. Potential
team members could include:
• Forester
• State forestry personnel
• Hydrologist
• Pesticide specialist
• Soil scientist
* Water quality expert
The composition of evaluation teams can
vary depending on the purpose of the
evaluation, available staff and other
resources, and the geographic area being
covered. All team members should be
familiar with the required MMs/BMPs, and
each team should have a member who has
previously participated in a site evaluation.
This will ensure familiarity with the
technical aspects of the MMs/BMPs that will
be rated during the evaluation and the site
evaluation process.
Training may be necessary to bring all team
members to the level of proficiency needed
to conduct the site evaluations. State
forestry personnel should be familiar with
forestry regulations, state BMP standards
and specifications, and proper BMP
implementation, and therefore are generally
well qualified to teach these topics to
evaluation team members who are less
familiar with them. This training should
include identification of BMPs particularly
critical to water quality protection, BMPs
implemented poorly in previous years,
analysis of erosion potential, and other
aspects of BMP implementation that require
professional judgement, as well as any
standard methods for measurements to judge
BMP implementation against state standards
and specifications.
Alternatively, if only one or two individuals
will be conducting site evaluations, their
training in the various specialties, such as
those listed above, necessary to evaluate the
quality of MM/BMP implementation could
be provided by a team of specialists who are
familiar with forestry practices and nonpoint
source pollution.
In the interest of consistency among the
evaluations and among team members, it is
advisable that one or more mock evaluations
take place prior to visiting selected sample
sites. These "practice sessions" provide
team members with an opportunity to
become familiar with MMs and BMPs as
they should be implemented under different
harvest site conditions, gain familiarity with
-------
Chapter 4
the evaluation forms and the meanings of the
terms and questions on them, and learn from
other team members with different expertise.
Mock evaluations are valuable for ensuring
that all evaluators have a similar
understanding of the intent of the questions,
especially for questions whose responses
involve a degree of subjectivity on the part
of the evaluator.
Where site evaluation teams are composed of
more than two or three people, it might be
helpful to divide the various responsibilities
for conducting the site evaluations among
team members ahead of time to avoid
confusion at the harvest site and to be certain
that all tasks are completed but not
duplicated. Having a spokesperson for the
group who is responsible for communicating
with the landowner or harvester—prior to the
site evaluation, at the site evaluation if they
are present, and afterward—might also be
helpful. A state forestry representative is
generally a good choice as spokesperson
because he/she represents the state forestry
authority. Newly formed evaluation teams
might benefit most from a division of labor
and selection of a team leader or team
coordinator with experience with site
evaluations who will be responsible for the
quality of the site evaluations. Smaller
teams might find that a division of
responsibilities is not necessary, as might
larger teams whose members have
experience working together. If
responsibilities are to be assigned, mock
evaluations can be a good time to work out
these details.
4.3.2 Rating Implementation of
Management Measures and Best
Management Practices
Many factors influence the implementation
of MMs and BMPs, so it is sometimes
necessary to use best professional judgment
(BPJ) to rate their implementation and BPJ
will almost always be necessary when rating
the implementation of MMs or when rating
overall BMP compliance at a harvest site.
Site-specific factors such as soil type, slope,
presence of a water body, and ground cover
type affect the implementation of erosion and
sediment control BMPs, for instance, and
must be taken into account by evaluators
when rating MM/BMP implementation.
Implementation of MMs will often be based
on implementation of more than one BMP,
and this makes rating MM implementation
similar to rating overall BMP
implementation at a harvest site.
Determining an overall rating involves
grouping the ratings of implementation of
individual BMPs into a single rating, which
introduces more subjectivity than rating the
implementation of individual BMPs based on
standards and specifications. Choice of a
rating system and rating terms, which are
aspects of proper evaluation design, is
therefore important in minimizing the level
of subjectivity associated with overall BMP
compliance and MM implementation ratings.
When creating overall ratings, it is still
important to record the detailed ratings of
individual BMPs as supporting information.
Individual BMPs, overall BMP compliance,
and MMs can be rated using a binary
approach (e.g., pass/fail, compliant/
noncompliant, or yes/no) or on a scale with
-------
ConUuethm the Evaluation
Chapter 4
Example... of a rating scale (Source:
Rossman and Phillips, 1992). More
examples are presented in Appendix B.
Minnesota Division of Forestry uses this 5-
choice rating scale for BMP implementation
audits:
5 = Operation exceeds requirement of BMP
4 = Operation meets requirement of BMP
3 « Minor departure from BMP
2 s Major departure from BMP
1'- Gross negtect of BMP
where;
Minor departure, is defined as- "small in
magnitude and localized,"major departure is
defined as "significant magnitude or where the
BMPs were consistently neglected" and gross
neglect is defined as "potential risk to water
resources was significant and there was no
evidence that any attempt has been made by
the operator to apply the BMP."
more than two choices, such as 1 to 5 or 1 to
10 (where 1 is the worst) (see Example).
The simplest method of rating MM and BMP
implementation is the use of a binary
approach. Using a binary approach, either
an entire site or individual MMs or BMPs
are rated as being hi compliance or not hi
compliance with respect to specified criteria.
Scale systems can take the form of ratings
from poor to excellent, inadequate to
adequate, low to high, 1 to 3, 1 to 5, and so
forth.
Whatever form of scale is used, the factors
that would individually or collectively
qualify a site, MM, or BMP for one of the
ratings should be clearly stated. The more
choices that are added to the scale, the
smaller and smaller the difference between
them becomes and each must therefore be
defined more specifically and accurately.
This is especially important if different teams
or individuals rate separate sites.
Consistency among the ratings then depends
on each team or individual evaluator
knowing precisely what the criteria for each
rating option mean. Clear and precise
explanations of the rating scale can also help
avoid or reduce disagreements among team
members. This applies equally to a binary
approach. The factors, individually or
collectively, that would cause a site, MM, or
BMP to be rated as not being hi compliance
with design specifications should be clearly
stated on the evaluation form or in
supporting documentation.
Rating sites or MMs/BMPs on a scale
requires a greater degree of analysis by the
evaluation team than does using a binary
approach. Each higher number represents a
better level of MM/BMP implementation
and/or effectiveness. In effect, a binary
rating approach is a scale with two choices; a
scale of low, medium, and high (compliance)
is a scale with three choices. Use of a scale
system with more than two rating choices
can provide more information to program
managers than a binary rating approach, and
this factor must be weighed against the
greater complexity involved hi using one.
For instance, a survey that uses a scale of 1
to 5 might result hi one MM with a rating of
1, five with a rating of 2, six with a rating of
3, eight with a rating of 4, and five with a
rating of 5. Precise criteria would have to
be developed to be able to ensure consistency
within and between survey teams hi rating
the MMs, but the information that only 1
MM was poorly implemented, 11 were
below standards, 13 met or were above
-------
Chapter 4
standards, and 5 were implemented very well
might be more valuable than the information
that 18 MMs were found to be in compliance
with design specifications, which is the only
information that would be obtained with a
binary rating approach.
If a rating system with more than two ratings
is used to collect data, the data can be
analyzed either by using the original rating
data or by first transforming the data into a
binomial (i.e., two-choice rating) system.
For instance, ratings of 1 through 5 could be
reduced to two ratings by grouping the Is,
2s, and 3s together into one group (e.g.,
inadequate) and the 4s and 5s into a separate
group (e.g., adequate). If ihis approach is
used, it is best to retain the rating data for
the detailed information it contains and to
reduce the data to a binomial system only for
the purpose of statistical analysis. Chapter
3, Section 3.5, contains information on the
analysis of categorical data.
4.3.3 Rating Terms
The choice of rating terms used on the
evaluation forms is an important factor in
ensuring consistency and reducing bias, and
the terms used to describe and define the
rating options should be as objective as
possible. For a rating system with a large
number of options, the meanings of each
option should be clearly defined. It is best
to avoid using terms such as "major" and
"minor" when describing erosion or
pollution effects or deviations from
prescribed MM/BMP implementation criteria
because they might have different
connotations for different evaluation team
members. It is easier for an evaluation team
to agree on meaning if options are described
in terms of measurable criteria and examples
are provided to clarify the intended meaning.
It is also best not to use terms that carry
negative connotations. Evaluators are less
likely to rate something as having a "major
deviation" from an implementation criterion,
even if justified, because of the negative
connotation carried by the term. Rather than
using such a term, observable conditions or
effects of the quality of implementation
should be listed and specific ratings (e.g., 1-
5 or compliant/noncompliant for the
criterion) should be associated with the
conditions or effects. For example, instead
of rating culvert installation as having a
"major deficiency," a specific deficiency
should be described and should have an
associated rating ascribed to it (e.g.,
"Culvert as installed does not allow for fish
passage = noncompliant").
Evaluation team members will often have to
take specific notes on sites, MMs, or BMPs
during the evaluation, either to justify the
ratings they have ascribed to variables or for
discussion with other team members after the
survey. When recording notes about the
sites, MMs, or BMPs, evaluation team
members should be as specific as the criteria
for the ratings. A rating recorded as "MM
deviates highly from implementation
criteria" is highly subjective and loses
specific meaning when read by anyone other
than the person who wrote the note. Notes
should therefore be as objective and specific
as possible.
An overall site rating is useful for
summarizing information in reports;
identifying the overall level of
-------
Conducting the Evaluation
Chapter 4
implementation with MMs/BMPs, indicating
the likelihood that environmental protection
is being achieved, identifying additional
training or education needs; and conveying
information to program managers who are
often not familiar with MMs or BMPs. For
the purposes of preserving the valuable
information contained in the original ratings
of sites, MMs, or BMPs, however, overall
ratings should summarize, not replace the
original data. Analysis of year-to-year
variations in MM or BMP implementation,
the factors involved hi MM or BMP program
implementation, and factors that could
improve MM or BMP implementation and
MM or BMP program success are possible
only if the original, detailed site, MM, or
BMP data are used.
Approaches commonly used for determining
final BMP implementation ratings include
calculating a percentage based on individual
BMP ratings, consensus, compilation of
aggregate scores by an objective party,
voting, and voting only where consensus on
a site or MM/BMP rating cannot be reached.
Not all systems for arriving at final ratings
are applicable to all circumstances.
4.3.4 Consistency Issues
Consistency among evaluators and between
evaluations is important. Consistency is
likely to be best if only one or two
evaluators conduct the site evaluations and
the same persons conduct all of the
evaluations. If, for statistical purposes,
many sites (e.g., 100 or more) need to be
evaluated, use of only one or two evaluators
might also be the most efficient approach.
In this case, a team of evaluators might be
useful for revisiting a subsample of the sites
evaluated by the one to two persons for
quality control purposes. Evaluation teams
can also be useful for training the one or two
persons who will conduct the site evaluations
in their specialties as they relate to
MM/BMP implementation and nonpoint
source pollution.
If teams of evaluators conduct the
evaluations, consistency can be achieved by
keeping the membership of the teams
constant. Differences of opinion, which are
likely to arise among team members, can be
settled through discussions held during
evaluations, and the experience of team
members who have done past evaluations can
help guide decisions. Pre-evaluation training
sessions, such as the mock evaluations
discussed above, will help ensure that the
first few site evaluations are not "learning"
experiences to such an extent that sites must
be revisited to ensure that they receive the
same level of scrutiny as sites evaluated
later.
If different sites are visited by different
teams of evaluators or if individual
evaluators are assigned to different sites, it is
especially important that consistency be
established before the evaluations are
conducted. For best results, discussions
among evaluators should be held periodically
during the evaluations to discuss any
potential problems. For instance, evaluators
could visit some sites together at the
beginning of the evaluations to promote
consistency in ratings, followed by site
evaluations conducted by individual
evaluators. Then, after a few site or MM
evaluations, evaluators could gather again to
-------
Chapter 4
discuss results and to share any knowledge
gained to ensure continued consistency.
As mentioned above, consistency can be
established during mock evaluations held
before the actual evaluations begin. These
mock evaluations are excellent opportunities
for evaluators to discuss the meaning of
terms on rating forms, differences between
rating criteria, and differences of opinion
about proper MM/BMP implementation. A
member of the evaluation team should be
able to represent the state's position on the
definition of terms and clarify areas of
confusion.
Descriptions of MMs and BMPs should be
detailed enough to support any ratings given
to individual features and to the MM or
BMP overall. Sketching a diagram of the
MM or BMP helps identify design problems,
promotes careful evaluation of all features,
and provides a record of the MM or BMP
for future reference. A diagram is also
valuable when discussing the MM or BMP
with the landowner or identifying features in
need of improvement or alteration.
Landowners can also use a copy of the
diagram and evaluation when discussing their
operations with state forestry agents.
Photographs of MM or BMP features are
valuable reference material and should be
used whenever an evaluator feels that a
written description or a diagram could be
inadequate. Photographs of what constitutes
both good and poor MM or BMP
implementation are valuable for explanatory
and educational purposes; for example, for
presentations to managers and the public.
4.3.5 Postevaluation Onsite Activities
It is important to complete all pertinent tasks
as soon as possible after the completion of a
site evaluation to avoid extra work later and
to reduce the chances of introducing error
attributable to memory error or confusion.
All evaluation forms for each site should be
filled out completely before leaving the site.
Information not filled in at the beginning of
the evaluation can be obtained from the
landowner if necessary. Any questions that
evaluators had about the MMs/BMPs during
the evaluation can be discussed, and notes
written during the evaluation can be shared
and used to help clarify details of the
evaluation process and ratings. The
opportunity to revisit the site will still exist
if there are points that cannot be agreed upon
among evaluation team members.
Also, while the evaluation team is still on
site, the landowner should be informed about
what will follow; for instance, whether
he/she will receive a copy of the report,
when to expect it, what the results means,
and his/her responsibility in light of the
evaluation, if any. Immediately following
the evaluation is also an excellent time to
discuss the findings with the landowner if
he/she was not present during the evaluation.
4.4 SELF-EVALUATIONS
4.4.1 Methods
Self-evaluations, while often not a reliable
source of MM or BMP implementation data,
can be used to augment data collected
through expert evaluations or in place of
expert evaluations where the latter cannot be
-------
Conducting the Evaluation
Chapter 4
conducted. In some cases, state forestry
authority staff might have been involved
directly with a harvest and will be a source
of useful information even if an expert
evaluation is not conducted. Self-evaluations
are an appropriate survey method for
obtaining background information from
harvesters and landowners.
Mail, telephone, and mail with telephone
follow-up are common self-evaluation
methods. Mail and telephone surveys are
useful for collecting general information,
such as the location of harvest operations,
species harvested, methods used, and dates
of harvest. Also, harvest application or
notification records can provide useful
background information, including any
special conditions applied to the harvest by
the state forestry authority. Recent advances
in and increasing access to electronic means
of communication (i.e., e-mail and the
Internet) might make these viable survey
instruments in the future.
Mail surveys with a telephone follow-up
and/or site visit are an efficient method of
collecting information. To ensure
comparability of results, information
collected as part of a self-
evaluation—whether collected through the
mail, over the phone, or during site
visits—should be collected in a manner that
does not favor one method over the others.
Ideally, telephone follow-up and site visit
interviews should consist of no more than
reading the questions on the questionnaire,
without providing any additional explanation
or information that would not have been
available to those who responded through the
mail. This approach eliminates as much as
possible any bias associated with the
different means of collecting the information.
Questionnaire design is discussed in Section
4.4.3.
It is important that the accuracy of
information received through mail and phone
surveys be checked. Inaccurate or
incomplete responses to questions on mail
and/or telephone surveys commonly result
from survey respondents misinterpreting
questions and thus providing misleading
information, not including all relevant
information hi their responses, not wanting
to provide some types of information, or
deliberately providing some inaccurate
responses. Therefore, the accuracy of
information received through mail and phone
surveys should be checked by selecting a
subsample of the harvesters and/or
landowners surveyed and conducting follow-
up site visits.
4.4.2 Cost
Cost can be an important consideration when
selecting an evaluation method. Site visits
can cost several hundred dollars per harvest
operation depending on the complexity of the
operation, the information to be collected,
and the number of evaluators used. Mail
and/or telephone surveys can be an
inexpensive means of collecting information,
but their cost must be balanced with the type
and accuracy of information that can be
collected through them. Other costs need to
be figured into the overall cost of mail
and/or telephone surveys as well, including
follow-up phone calls and site visits to make
up for a poor response to mailings and for
accuracy checks. Additionally, the cost of
-------
Chapter 4
f-^ , ; .i,. ;m;tl™s-!r' ™"l "««fsl=*
Conducting the Evaluation
questionnaire design must be considered, as
a well-designed questionnaire is extremely
important to the success of self-evaluations.
Questionnaire design is discussed in the next
section.
The number of evaluators used for site visits
has an obvious impact on the cost of a
MM/BMP implementation survey. Survey
costs can be minimized by having one or two
evaluators visit harvest sites instead of
having multiple-person teams visit each
survey site. If the expertise of many
specialists is desired, it might be cost-
effective to have multiple-person teams
check the quality of evaluations conducted by
one or two evaluators. This can usually be
done at a subsample of harvest sites after the
sites have been surveyed.
An important factor to consider when
determining the number of evaluators to
include on site visitation teams, and how to
balance the use of one to two evaluators
versus multiple-person teams, is the
objectives of the survey. Cost
notwithstanding, the teams conducting the
site evaluations must be sufficient to meet
the objectives of the survey, and if the
required teams would be too costly, then the
objectives of the survey would need to be
modified.
Another factor that contributes to the cost of
a MM/BMP implementation survey is the
number of sites to be surveyed. Once again,
a balance must be reached between cost, the
objectives of the survey, and the number of
sites to be evaluated. Generally, once the
objectives of the study have been specified,
the number of sites to be evaluated is
determined statistically to meet required data
quality objectives. If the number of sites
that is determined in this way would be too
costly, then it would be necessary to modify
the study objectives or the data quality
objectives. Statistical determination of the
number of sites to evaluate is discussed in
Section 2.3.
4.4.3 Questionnaire Design
Many books have been written on the design
of data collection forms and questionnaires
(e.g., Churchill, 1983; Ferberetal., 1964;
Tull and Hawkins, 1990), and these can
provide good advice for the design of simple
questionnaires to be used for a single survey.
However, for complex questionnaires or
ones that will be used for initial surveys as
part of a series of surveys (i.e., trend
analysis), it is strongly advised that a
professional in questionnaire design be
consulted. Although it might seem that
designing a questionnaire is a simple task,
small details such as the order of questions,
the selection of one word or phrase over a
similar one, and the tone of the questions can
significantly affect survey results. A
professionally designed questionnaire can
yield information beyond that contained in
the responses to the questions themselves,
while a poorly designed questionnaire can
invalidate the results.
The objective of a questionniare, which
should be closely related to the objectives of
the survey, should be extremely well thought
out prior to its being designed.
Questionnaires should also be designed at the
same time as the information to be collected
is selected to ensure that the questions
-------
Conducting the Evaluation
Chapter 4
address the objectives as precisely as
possible. Conducting these activities
simultaneously also provides immediate
feedback on the attainability of the objectives
and the level of detail of information that can
be collected. For example, a researcher
might want information on protection of
habitat near surface waters, but might
discover while designing the questionnaire
that the desired information cannot be
obtained through the use of a questionnaire,
or that the information that could be
collected would be insufficient to fully
address the chosen objectives. In such a
situation the researcher could revise the
objectives and questions before going further
with questionnaire design.
Tull and Hawkins (1990) identified seven
major elements of questionnaire
construction:
1. Preliminary decisions
2. Question content
3. Question wording
4. Response format
5. Question sequence
6. Physical characteristics of the
questionnaire
7. Pretest and revision
Preliminary decisions .include determining
exactly what type of information is required,
determining the target audience, and
selecting the method of communication (e.g,
mail, telephone, site visit). These subjects
are addressed hi other sections of this
guidance.
The second step is to determine the content
of the questions. Each question should
generate one or more of the information
requirements identified in the preliminary
decisions. The ability of the question to
produce the necessary data needs to be
assessed. "Double-barreled" questions, in
which two or more questions are asked as
one, should be avoided. Questions that
require the respondent to aggregate several
sources of information should be subdivided
into several specific questions or parts. The
ability of the respondent to answer accurately
should also be considered when preparing
questions. Some respondents might be
unfamiliar with the type of information
requested or the terminology used. A
respondent might have forgotten some of the
information of interest, or might be unable to
verbalize an answer. Consideration should
be given to the willingness of respondents to
answer the questions accurately. If a
respondent feels that a particular answer
might be embarrassing or personally harmful
(e.g., might lead to fines or increased
regulation), he or she might refuse to answer
the question or might deliberately provide
inaccurate information. For this reason,
answers to questions that might lead to such
responses should be checked for accuracy
whenever possible.
The next step is to decide on the specific
phrasing of the questions. Simple, easily
understood language is preferred. The
wording should not bias the answer or be too
subjective. For instance, a question should
not ask whether groundskidding led to
erosion during the harvest. Instead, a series
of questions could ask whether
groundskidding was used, the slope of land
on which it was used, which BMPs were
used initially to control erosion from
-------
Chapter 4
groundskidding, and what additional
measures were used to control erosion from
groundskidding (if erosion occurred). These
questions all request factual information of
which a forest operator should be
knowledgeable, and the questions progress
from simple to more complex. All
alternatives and assumptions should be
clearly stated on the questionnaire, and the
respondent's frame of reference should be
considered.
Fourth, the type of response format should
be selected. Various types of information
can best be obtained using open-ended,
multiple-choice, or dichotomous questions.
An open-ended question allows respondents
to answer in any way they feel is
appropriate. Multiple-choice questions tend
to reduce some types of bias and are easier
to tabulate and analyze; however, good
multiple-choice questions can be more
difficult to formulate. Dichotomous
questions allow only two responses, such as
"yes-no" or "agree-disagree." Dichotomous
questions are suitable for determining points
of fact, but they must be very precisely
stated and unequivocally solicit only a single
piece of information.
The fifth step in questionnaire design is the
ordering of the questions. The first
questions should be simple to answer,
objective, and interesting in order to relax
the respondent. The questionnaire should
move from topic to topic in a logical manner
without confusing the respondent. Early
questions that could bias the respondent
should be avoided. There is evidence that
response quality declines near the end of a
long questionnaire (Tull and Hawkins,
1990). Therefore, more important
information should be solicited early.
Before presenting the questions, the
questionnaire should explain how long (on
average) it will take to complete and the
types of information that will be solicited.
The questionnaire should not present the
respondent with any surprises.
The layout of the questionnaire should make
it easy to use and should minimize recording
mistakes. The layout should clearly show
the respondent all possible answers. For
mail surveys, an attractive appearance is
important for securing cooperation.
The final step in the design of a
questionnaire is the pretest and possible
revision. A questionnaire should always be
pretested with members of the target
audience. This will preclude expending
large amounts of effort and then discovering
that the questionnaire produces biased or
incomplete information.
4.5 AERIAL RECONNAISSANCE AND
PHOTOGRAPHY
Aerial reconnaissance and photography can
be useful tools for gathering harvest site
information quickly and comparatively
inexpensively. For the purposes of forestry
BMP compliance surveying, aerial
reconnaissance can be useful for selecting
survey sites and evaluating some aspects of
harvest sites. In Florida, survey sites for
each county are selected from fixed-wing
aircraft flown in a predetermined pattern.
This approach reduces bias in selecting
survey sites. The selected sites are then
visited by foresters for BMP compliance
-------
Conducting the Evaluation
Chapter 4
evaluations (Vowell and Gilpin, 1994).
Survey sites are selected from fixed-wing
aircraft in South Carolina as well (Adams,
1994). In addition, aerial photography has
been proven to be helpful for forest
regeneration assessment (Hall and Aldred,
1992; Hudson, 1988); forest inventory and
analysis (Hackett, 1988); terrain
stratification, riparian area delineation,
vegetation mapping, stream morphology
characterization, inventory site identification,
planning, and monitoring in mountainous
regions (Born and Van Hooser, 1988;
Hetzel, 1988); rangeland monitoring (BLM,
1991); and agricultural conservation practice
identification (Pelletier and Griffin, 1988).
Factors such as the characteristics of what is
being monitored, scale, and camera format
determine how useful aerial photography can
be for a particular purpose.
Photographic scale and resolution must be
taken into consideration when deciding
whether to use aerial photography, and a
photographic scale that produces good
resolution of the items of importance to the
monitoring effort must be chosen. Born and
Van Hooser (1988), investigating the
usefulness of aerial photography for the
classification of inventory and monitoring
sample points and locating the sample points
on the ground, found that a scale of 1:58,000
(i.e., 1 unit on a photograph represents
58,000 units on the ground) was marginal
for use in forestry resource inventorying and
monitoring. Hetzel (1988) and Mereszczak
(1988), using a large-format camera (see
below), found that at a scale of 1:30,000
riparian areas were easily distinguishable and
could be delineated with 100 percent
accuracy, and cover types could be
delineated with 83 percent accuracy.
Mereszczak (1988) found that aerial
photography was especially useful for
monitoring riparian areas because changes in
their ecological condition in response to
management practices are evident over time
frames of 10 years or less. Reutebuch and
Shea (1988) reported that photographs taken
at a scale of 1:12,000 or larger have a
typical resolution of less than 1 foot on the
ground. Hall and Aldred (1992) were able
to clearly delineate and map cutovers and
nonforest areas (water bodies, roads,
landings, clearings, brush areas) at a
photographic scale of 1:10,000, and could
detect conifer seedlings 30 cm or taller, if
not hidden beneath other trees, at scales of
1:800 to 1:500. The Bureau of Land
Management (BLM) uses low-level, large-
scale (1:1,000 to 1:1,500) aerial photography
to monitor rangeland vegetation (BLM,
1991). BLM reports that scales smaller than
1:1,500 (e.g., 1:10,000, 1:30,000) are too
small to monitor the classes of land cover
(shrubs, grasses and forbs, bare soil, rock)
on rangeland.
Pelletier and Griffin (1988) investigated the
use of aerial photography for the
identification of agriculture conservation
practices. They found that practices that
occupy a large area and have an identifiable
pattern, such as contour cropping, strip
cropping, terraces, and windbreaks, were
readily identified even at a small scale
(1:80,000) but that smaller, single-unit
practices, such as sediment basins and
sediment diversions, were difficult to
identify at a small scale. They estimated that
29 percent of practices could be identified at
a scale of 1:80,000, 45 percent could be
-------
Chapter 4
Conducting "tfie • EvaluaKd
'
identified at 1:30,000, 70 percent could be
identified at 1:15,000, and over 90 percent
could be identified at a scale of 1:10,000.
Camera format is another factor that must be
considered. Large-format cameras are
generally preferred over small-format
cameras (e.g., 35mm), but are more costly
to purchase and operate. The large negative
size (9 cm x 9 cm) produced using a large-
format camera provides the resolution and
detail necessary for accurate photo
interpretation. Large-format cameras can be
used from higher altitudes than small-format
cameras, and the image area covered by a
large-format image at a given scale (e.g.,
1:1,500) is much larger than the image area
captured by a small-format camera at the
same scale. Small-format cameras can be
used for identifications that involve large-
scale features, such as mining areas, the
extent of burning, and large animals in
censuses, and they have definite applications
in forestry as well. Owens (1988)
recommends 35mm photography for
monitoring small areas (^3 mi2) at low
altitude. A particularly useful application of
35mm photography is mapping private
landowner parcels and change monitoring
(Owens, 1988). Owens (1988) used large-
format photographs as baseline data and in
subsequent years used 35mm photographs to
monitor timber harvests with much success.
Small-format cameras are limited in the
resolution that they provide when
photographs are enlarged (Owens, 1988).
BLM recommends the use of a large-format
camera because it provides flexibility to
increase sample plot size, it permits modest
navigational errors during overflight, and the
images provide the photo interpreter with
more geographical reference points (BLM,
1991). Large-scale photographs have
advantages over topographic maps.
Specifically, they have much higher
resolution, contain many more features and
ground characteristics, and—when viewed in
stereo—provide an accurate, 3-dimensional
model of an area, complete with vegetative
cover information and land-use
characteristics (Reutebuch and Shea, 1988).
Also, large-format photography equipment is
standard equipment for most photo
contractors, so one could be hired to take the
photographs in lieu of purchasing the
equipment.
A drawback to the use of aerial photography
is that forestry BMPs that do not meet
implementation or operational standards but
are similar to practices that do are
indistinguishable from ones that do in an
aerial photograph (Pelletier and Griffin,
1988). Also, practices that are defined by
managerial concepts rather than physical
criteria, such as preharvest planning or forest
chemical management, cannot be detected
with aerial photographs.
Regardless of scale, format, or item being
monitored, it is useful for photo interpreters
to receive 2 or 3 days of training on the
basic fundamentals of photo interpretation
and important that they be thoroughly
familiar with the vegetation and landforms in
the areas where the photographs that they
will be interpreting were taken (BLM,
1991). A site visit to the field sites in the
photographs is recommended to improve
correlation between the interpretation and
actual site characteristics. Usually, after a
-------
Conducting the Evaluation
Chapter 4
few site visits and interpretations of
photographs of those sites, photo interpreters
will be familiar with the photographic
characteristics of the vegetation hi the area
and site visits can be reserved for
verification of items hi doubt. A change in
type of vegetation or physiography hi
photographs normally requires new site visits
until photo interpreters are familiar with the
characteristics of the new vegetation hi the
photographs.
-------
CHAPTER 5. PRESENTATION OF EVALUATION RESULTS
5.1 INTRODUCTION
The first three chapters of this guidance
presented techniques for the collection of
information. Data analysis and interpretation
are addressed in detail in Chapter 8 of EPA's
Nonpoint Source Monitoring and Evaluation
Guide (USEPA, 1996). This chapter
provides ideas for the presentation of results.
The presentation of MM/BMP compliance
evaluation results, whether written or oral, is
an integral part of a successful compliance
survey. The quality of the presentation of
results is an indication of the quality of the
survey, and if the presentation fails to
convey important information from the
compliance survey to those who need the
information, the compliance survey itself
might be considered a failure.
The quality of the presentation of results is
dependent on at least four criteria—results
must be complete, accurate, clear, and
concise (Churchill, 1983). Completeness
means that the presentation provides all
necessary information to the audience in
language that it understands; accuracy is
determined by how well a researcher handles
the data, phrases findings, and reasons;
clarity is the result of clear and logical
thinking and a precision of expression; and
conciseness is the result of selecting for
inclusion only that which is necessary.
Throughout the process of preparing the
results of an MM/BMP compliance survey
for presentation, it must be kept in mind that
the survey was initially undertaken to
provide information for management
purposes—specifically, to help make a
decision (Tull and Hawkins, 1990). The
presentation of results should be built around
the decision that the compliance survey was
undertaken to support. The message of the
presentation must also be tailored to that
decision. It must be realized that there will
be a time lag between the compliance survey
and presentation of the results, and the
results should be presented in light of their
applicability to the management decision to
be made based on them. The length of the
time lag is a key factor in determining this
applicability. If the time lag is significant, it
should be made clear during the presentation
that the situation might have changed since
the survey was conducted. If reliable trend
data are available, the person making the
presentation might be able to provide a sense
of the likely magnitude of any change in the
situation. If the change in status is thought
to be insignificant, evidence should be
presented to support this claim. For
example, state that "At the time that the
compliance survey was conducted, forest
harvesters were using BMPs with increasing
frequency, and the lack of any changes in
program implementation coupled with
continued interaction with forest harvesters
provides no reason to believe that this trend
has changed since that time." It would be
misleading to state "The monitoring study
indicates that forest harvesters are using
BMPs with increasing frequency."* The
validity and force of the message will be
enhanced further through use of the active
voice (we believe) rather than the passive
voice (it is believed).
-------
Presentation of Evaluation Results
Chapter 5
Three major factors must be considered
when presenting the results of MM and BMP
implementation studies: (1) identifying the
target audience, (2) selecting the appropriate
medium (printed word, speech, pictures,
etc.), and (3) selecting the most appropriate
format to meet the needs of the audience.
5.2 AUDIENCE IDENTIFICA TION
Identification of the audience(s) to which the
results of the MM and BMP implementation
study will be presented determines the
content and format of the presentation. For
the results of implementation monitoring
studies, there are typically six potential
audiences:
Interested/concerned citizens
Forest land owners and
harvesters
Media/general public
Policy makers
Resource managers
Scientists
•
•
*
•
These audiences have different information
needs, interests, and abilities to understand
complex data. It is the job of the person(s)
preparing the presentation to analyze these
factors prior to preparing a presentation.
The four criteria for presentation quality
apply regardless of the audience. Other
elements of a comprehensive presentation,
such as discussion of the objectives and
limitations of the study and necessary details
of the method, must be part of the
presentation and must be tailored to the
audience. For instance, details of the
sampling plan, why the plan was chosen over
others, and the statistical methods used for
analysis should be recorded even if they are
not part of any presentation of results
because of their value for future reference
when the monitoring is repeated or similar
studies are undertaken, but they are best not
included in a presentation to management.
5.3 PRESENTA TION FORMA T
Regardless of whether the results of a
monitoring study are presented in writing or
orally, or both, the information being
presented must be understandable to the
audience. Consideration of who the
audience is will help ensure that the
presentation is particularly suited to the
audience's needs. Selection of the correct
format for the presentation will ensure that
the information is conveyed in a manner that
is easy to comprehend.
Most reports will have to be presented in
both a written and an oral form. Written
reports are valuable for peer review, public
information dissemination, and future
reference. Oral presentations are often
necessary for managers, who usually do not
have tune to read an entire report, have need
for only the results of the study, and are
usually not interested in the finer details of
the study. Different versions of a
report—for the public, scientists, and
managers (i.e., an executive
summary)—might well have to be written,
and separate oral presentations for different
audiences—the public, farmers, managers,
and scientists at a conference—might have to
be prepared.
Most information can most effectively be
presented in the form of tables, charts, and
-------
Chapter 5
diagrams (Tull and Hawkins, 1990). These
graphic forms of data and information
presentation can help simplify the
presentation, making it easier for an
audience to comprehend than if explained
exhaustively with words. Words are
important for pointing out significant ideas
or findings, and for interpreting the results
where appropriate. Words should not be
used to repeat what is already adequately
explained in graphics, and slides or
transparencies that are composed largely of
words should contain only a few essential
ideas each. Presentation of too much written
information on a single slide or transparency
only confuses the audience. Written slides
or transparencies should also be free of
graphics, such as clever logos or background
highlights—unless the pictures are essential
to understanding the information
presented—since they only make the slides
or transparencies more difficult to read.
Examples of graphics and written slides are
presented in Figures 4-1 through 4-3.
Different types of graphics have different
uses as well. Information presented in a
tabular format can be difficult to interpret
because the reader has to spend some time
with the information to extract the essential
points from it. The same information
presented in a pie chart or bar graph can
convey essential information immediately
and avoid the inclusion of background data
that are not essential to the point. When
preparing information for a report, a
researcher should organize the information in
various ways and choose that which conveys
only the information essential for the
audience in the least complicated manner.
5.3.1 Written Presentations
The following criteria should be considered
when preparing written material:
• Reading level or level of
education of the target
audience.
• Level of detail necessary to
make the results
understandable to the target
audience. Different audiences
require various levels of
background information to
fully understand the results of
a study.
• Layout. The integration of
text, graphics, color, white
space, columns, sidebars, and
other design elements is
important in the production of
material that the target
audience will find readable
and visually appealing.
• Graphics. Photos, drawings,
charts, tables, maps, and other
graphic elements can be used
to effectively present
information that the reader
might otherwise not
understand.
5.3.2. Oral Presentations
An effective oral presentation requires
special preparation. Tull and Hawkins
(1990) recommend three steps:
-------
Presentation of Evaluation Results
Chapter 5
Siwtlmfasr Polsilmbsr Buud-
••pllng
Non-
stocked
Esst
W»«t
S*sd-»pllng
23%
East
Nonstocksd
1%
Pot*tlmb*r
21%
Sswtlmbsr
48%
West
Nonstockcd
2%
S»»d-*apllng
13%
Polctlmbsr
16%
S>wtlmb*r
70%
Figure 5-1. Timberland area by stand size class, East and West, 1992, represented graphically
in three ways. (After Powell et al., 1994)
-------
Chapter 5
Presentation of Evalvia^l^Sg
Other type groups
WhIte-red-Jack pine
Longleaf-slash pine
Elm-ash-cottonwood
Aspen-birch
Spruce-fir
Oak-gum-cypress
Oak-pine
Maple-beech-blrch
Loblolly-shortleaf pine
Oak-hickory
Million acres
80 100 120 140
Figure 5-2. Forest type groups on unreserved forest land in the
East, 1992, represented graphically. (After Powell et al., 1994)
FOREST LAND AREA
Increased 0.1% between 1987 and 1992
33% of U.S. land area (737 million acres)
Clearing forests for agriculture largely
halted by 1920
34% is federally owned
6% is reserved from commercial harvest
(47 million acres)
Figure 5-3. Example written presentation slide.
-------
Presentation of Evaluation Results
Chapter 5
1. Analyze the audience, as explained
above.
2. Prepare an outline of the
presentation, and preferably a written
script.
3. Rehearse the presentation. Several
dry runs should be made, and if
possible the presentation should be
taped on a VCR and analyzed.
These steps are extremely important if an
oral presentation is to be effective.
Remember that oral presentations of 1A to 1
hour are often all that is available for the
presentation of the results of months of
research to managers who are poised to
make decisions based on the presentation.
Adequate preparation is essential if the oral
presentation is to accomplish its purpose.
5.4 FOR FURTHER INFORMATION
Providing specific examples of effective and
ineffective presentation graphics, writing
styles, and methods of organization is
beyond the scope of this document. A
number of resources that contain suggestions
for how study results should be presented are
available, however, and should be consulted.
A listing of some references is provided
below.
• The New York Public Library
Writer's Guide to Style and
Usage (NYPL, 1994) has
information on design, layout,
and presentation in addition to
guidance on grammar and
style.
Good Style: Writing for
Science and Technology
(Kirkman, 1992) provides
techniques for presenting
technical material hi a
coherent, readable style.
The Modern Researcher
(Barzun and Graff, 1992)
explains how to turn research
into readable, well-organized
writing.
Writing with Precision: How
to Write So That You Cannot
Possibly Be Misunderstood,
6th ed. (Bates, 1993)
addresses communication
problems of the 1990s.
Designer's Guide to Creating
Chans & Diagrams (Holmes, ,
1991) gives tips for combining
graphics with statistical
information.
The Elements of Graph Design
(Kosslyn, 1993) shows how to
create effective displays of
quantitative data.
-------
REFERENCES
Academic Press. 1992. Dictionary of science and technology. Academic Press, Inc., San
Diego, California.
Adams, T. 1994. Implementation monitoring of forestry best management practices on
harvested sites in South Carolina. Best Management Practices Monitoring Report BMP-2.
South Carolina Forestry Commission, Columbia, South Carolina. September.
Barzun, J., and H.F. Graff. 1992. The modern researcher. 5th ed. Houghton Mifflin.
Bates, J. 1993. Writing with precision: How to write so thatyou cannot possibly be
misunderstood. 6th ed. Acropolis.
Blalock, H.M., Jr. 1979. Social statistics. Rev. 2nd ed. McGraw-Hill Book Company, New
York, NY.
BLM. 1991. Inventory and monitoring coordination: Guidelines for the use of aerial
photography in monitoring. Technical Report TR 1734-1. Department of the Interior, Bureau
of Land Management.
Born, J.D., and D.D. Van Hooser. 1988. Intermountain Research Station remote sensing
use for resource inventory, planning, and monitoring. In Remote sensing for resource
inventory, planning, and monitoring. Proceedings of the Second Forest Service Remote
Sensing Applications Conference, Sidell, Louisiana, and NSTL, Mississippi, April 11-15,
1988.
Casley, D.J., and D.A. Lury. 1982. Monitoring and evaluation of agriculture and rural
development projects. The Johns Hopkins University Press, Baltimore, MD.
Churchill, G. A., Jr. 1983. Marketing research: Methodological foundations, Srded. The
Dryden Press, New York, New York.
Cochran, W.G. 1977. Sampling techniques. 3rd ed. John Wiley and Sons, New York, New
York.
Colla, J. 1994. 1993 forest practices evaluation report. Report No. 2-94 FPA. State of
Idaho, Department of Lands.
Conover, W.J. 1980. Practical nonparametric statistics, 2nd ed. Wiley, New York.
-------
References
Cross-Smiecinski, A., and L.D. Stetzenback. 1994. Quality planning for the life science
researcher: Meeting quality assurance requirements. CRC Press, Boca Raton, Florida.
Curtis, J.G., D.W. Peken, D.B. George, V.D. Adams, and J.B. Layzer. 1990. Effectiveness
of best management practices in preventing degradation of streams caused by silvicultural
activities in Pickett State Forest, Tennessee. Tennessee Department of Conservation, Division
of Forestry, Nashville, Tennessee, and Tennessee Wildlife Resources Agency, Nashville,
Tennessee. December.
Dissmeyer, G.E. 1994. Evaluating the effectiveness of forestry best management practices in
meeting water quality goals or standards. Miscellaneous Publication 1520. U.S. Department
of Agriculture, Forest Service. July.
Ehinger, W., and D. Potts. 1991. On-site assessment of best management practices as an
indicator of cumulative watershed effects in the Flathead Basin. Flathead Basin Water Quality
and Fisheries Cooperative, University of Montana, School of Forestry, Missoula, Montana.
Ferber, R., D.F. Blankertz, and S. Hollander. 1964. Marketing research. The Ronald Press
Company, New York, NY.
*.
Freund, I.E. 1973. Modern elementary statistics. Prentice-Hall, Englewood Cliffs, New
Jersey. t
Gaugush, R.F. 1987. Sampling design for reservoir water quality investigations. Instruction
Report E-87-1. Department of the Army, US Army Corps of Engineers, Washington, DC.
Gilbert, R.O. 1987. Statistical methods for environmental pollution monitoring. Van Nostrand
Reinhold, New York, NY.
Hackett, R.L. 1988. Remote sensing at the North Central Forest Experiment Station. In
Remote sensing for resource inventory, planning, and monitoring. Proceedings of the Second
Forest Service Remote Sensing Applications Conference, Sidell, Louisiana, and NSTL,
Mississippi, April 11-15, 1988.
Hall, R.J., and A.H. Aired. 1992. Forest regeneration appraisal with large-scale aerial
photographs. The Forestry Chronicle 68(1): 142-150.
Helsel, D.R., and R.M. Hirsch. 1995. Statistical methods in water resources. Elsevier.
Amsterdam.
-------
Hetzel, G.E. 1988. Remote sensing applications and monitoring in the Rocky Mountain
region. In Remote sensing for resource inventory, planning, and monitoring. Proceedings of
the Second Forest Service Remote Sensing Applications Conference, Sidell, Louisiana, and
NSTL, Mississippi, April 11-15, 1988.
Holmes, N. 1991. Designer's guide to creating charts & diagrams. Watson-Guptill.
Hook, D., W. McKee, T. Williams, B. Baker, L. Lundquist, R. Martin, and J. Mills. 1991. A
survey of voluntary compliance of forestry BMPs. South Carolina Forestry Commission,
Columbia, South Carolina.
Hudson, W.D. 1988. Monitoring the long-term effects of silvicultural activities with aerial
photography. J. Forestry (March):21-26.
IDDHW. 1993. Forest practices water quality audit 1992. Idaho Department of Health and
Welfare, Division of Environmental Quality, Boise, Idaho.
Kirkman, J. 1992. Good style: Writing for science and technology. Chapman and Hall.
Kosslyn, S.M. 1993. The elements of graph design. W.H. Freeman.
Kupper, L.L., and K.B. Hafner. 1989. How appropriate are popular sample size formulas?
Am. Stat. 43:101-105.
MacDonald, L.H., A.W. Smart, and R.C. Wissmar. 1991. Monitoring guidelines to evaluate
the effects of forestry activities on streams in the Pacific Northwest and Alaska. EPA/910/9-91-
001. U.S. Environmental Protection Agency Region 10, Seattle, WA.
Mann, H.B., and D.R. Whitney. 1947. On a test of whether one of two random variables is
stochastically larger than the other. Annals of Mathematical Statistics 18:50-60.
McNew, R.W. 1990. Sampling and Estimating Compliance with BMPs. In Workshop on
implementation monitoring of forestry best management practices, Southern Group of State
Foresters, USDA Forest Service, Southern Region, Atlanta, GA, January 23-25, 1990, pp. 86-
105.
Mereszczak, I. 1988. Applications of large format camera - color infrared photography to
monitoring vegetation management withing the scope of forest plans. In Remote sensing for
resource inventory, planning, and monitoring. Proceedings of the Second Forest Service
Remote Sensing Applications Conference, Sidell, Louisiana, and NSTL, Mississippi, April 11-
15, 1988.
-------
References
NYPL. 1994. The New York public library writer's guide to style and usage. A Stonesong
Press book. HarperCollins Publishers, New York, NY.
Owens, T. 1988. Using 35mm photographs in resource inventories. In Remote sensing for
resource inventory, planning, and monitoring. Proceedings of the Second Forest Service
Remote Sensing Applications Conference, Sidell, Louisiana, and NSTL, Mississippi, April 11-
15, 1988.
Pelletier, R.E., and R.H. Griffin. 1988. An evaluation of photographic scale in aerial
photography for identification of conservation practices. /. Soil Water Conserv. 43(4):333-
337.
Powell, D.S., J.L. Faulkner, D.R. Darr, Z. Zhu, and D.W. MacCleery. 1994. Forest
resources of the United States, 1992. General Technical Report RM-234 (Revised). U.S.
Department of Agriculture, Forest Service. Rocky Mountain Forest and Range Experiment
Station, Fort Collins, Colorado. June.
Rashin, E., C. Clishe, and A. Loch. 1994. Effectiveness of forest road and timber harvest
best management practices with respect to sediment-related water quality impacts. Interim
Report No. 2. Washington State Department of Ecology, Environmental Investigations and
Laboratory Services program, Wastershed Assessments Section. Ecology Publication No. 94-
67. Olympia, Washington.
Remington, R.D., and M.A. Schork. 1970. Statistics with applications to the biological and
health sciences. Prentice-Hall, Englewood Cliffs, New Jersey.
Reutebuch, S.E., and R.D. Shea. 1988. A method to control large-scale aerial photos when
surveyed ground control is unavailable. In Remote sensing for resource inventory, planning,
and monitoring. Proceedings of the Second Forest Service Remote Sensing Applications
Conference, Sidell, Louisiana, and NSTL, Mississippi, April 11-15, 1988.
Rossman, R., and M.J. Phillips. 1991. Minnesota forestry best management practices
implementation monitoring. 1991 forestry field audit. Minnesota Department of Natural
Resources, Division of Forestry.
Schultz, B. 1992. Montana forestry best management practices implementation monitoring.
The 1992 Forestry BMP Audits Final Report. Montana Department of State Lands, Forestry
Division, Missoula, MT.
-------
References
Snedecor, G.W. and W.G. Cochran. 1980. Statistical methods. 7th ed. The Iowa State
University Press, Ames, Iowa.
Tull, D.S., and D.I. Hawkins. 1990. Marketing research. Measurement and method. Fifth
edition. Macmillan Publishing Company, New York, New York.
USDA. 1992. Investigating water quality in the pacific southwest region. BMPEP. Best
management practices evaluation program: A user's guide. U.S. Department of Agriculture,
Forest Service, Region 5, San Francisco, California. May.
USEPA. 1993a. Guidance specifying management measures for sources of nonpoint pollution
in coastal waters. EPA 840-B-92-002. U.S. Environmental Protection Agency, Office of
Water, Washington, DC.
USEPA. 1993b. Evaluation of the experimental rural clean water program. EPA 841-R-93-
005. U.S. Environmental Protection Agency, Office of Water, Washington, DC.
USEPA. 1995. National water quality inventory: 1994 report to Congress. EPA841-R-95-
005. U.S. Environmental Protection Agency, Office of Water, Washington, DC. December.
USEPA. 1996. Nonpoint source monitoring and evaluation guide. Final review draft. U.S.
Environmental Protection Agency, Office of Water, Washington, DC. November.
Vowell, L, and T. Gilpin. 1994. Florida silviculture best management practices compliance
survey report 1993. Florida Department of Agriculture and Consumer Services, Division of
Forestry, Tallahassee, Florida. May.
Wilcoxon, F. 1945. Individual comparisons by ranking metheds. Biometrics 1:80-83.
Winer, B.J. 1971. Statistical principles in experimental design. McGraw-Hill Book
Company, New York.
-------
-------
GLOSSARY
accuracy: the extent to which a measurement approaches the true value of the measured
quantity
aerial photography: the practice of taking photographs from an airplane, helicopter, or other
aviation device while it is airborne
allocation, Neyman: stratified sampling in which the cost of sampling each stratum is in
proportion to the size of the stratum but variability between strata changes
allocation, proportional: stratified sampling in which the variability and cost of sampling for
each stratum are in proportion to the size of the stratum
allowable error: the level of error acceptable for the purposes of a study
ANOVA: see test, analysis of variance
assumptions: characteristics of a population of a sampling method taken to be true without
proof
bar graph: a representation of data wherein data is grouped and represented as vertical or
horizontal bars over an axis
best professional judgement: an informed opinion made by a professional in the appropriate
field of study or expertise
best management practice: a practice or combination of practices that are determined to be the
most effective and practicable means of controlling point and nonpoint pollutants at levels
compatible with environmental quality goals
bias: a characteristic of samples such that when taken from a population with a known
parameter, their average does not give the parametric value
binomial: an algebraic expression that is the sum or difference of two terms
camera format: refers to the size of the negative taken by a camera. 35mm is a small camera
format
chi-square distribution: a scaled quantity whose distribution provides the distribution of the
sample variance
-------
Glossary
coefficient of variation', a statistical measure used to compare the relative amounts of variation
in populations having different means; the standard deviation divided by the mean
confidence interval: a range of values about a measured value hi which the true value is
presumed to lie
consistency: conformmg to a regular method or style; an approach that keeps all factors of
measurement similar from one measurement to the next to the extent possible
cumulative effects: the total influences attributable to numerous individual influences
degrees of freedom', the number of residuals (the difference between a measured value and the
sample average) required to completely determine the others
design, balanced: an ANOVA where all cells have equal numbers of samples
distribution: the allocation or spread of values of a given parameter among its possible values
e-mail: an electronic system for correspondence
erosion potential: a measure of the ease with which soil can be carried away in storm water
runoff or irrigation runoff
error: the fluctuation that occurs from one repetition to another; also experimental error
estimate, baseline: an appraisal of initial, or actual conditions
estimate, pooled: a single estimate obtained from grouping individual estimates and using the
latter to obtain a single value
finite population correction term: a correction term used when population size is small
relative to sample size
hydrologic modification: the alteration of the natural circulation or distribution of water by the
placement of structures or other activities
hypothesis, alternative: the hypothesis which is contrary to the null hypothesis
Jjypothesis, null: the hypothesis or conclusion assumed to be true prior to any analysis
Internet: an electronic data transmission system
-------
management measure: an economically achievable measure for the control of the addition of
pollutants from existing and new categories and classes of nonpoint sources of pollution,
which reflect the greatest degree of pollutant reduction achievable through the application of
the best available nonpoint pollution control practices, technologies, processes, siting criteria,
operating methods, or other alternatives
mean, estimated: a value of population mean arrived at through sampling
mean, overall: the measured average of a population
mean, stratum: the measured average within a sample subgroup or stratum
measurement bias: a consistent under- or overestimation of the true value of something being
measured, often due to the method of measurement
measurement error: the deviation of a measurement from the true value of that which is being
measured
median: the value of the middle term when data are arranged in order of size; a measure of
central tendency
monitoring, baseline: monitoring conducted to establish initial knowledge about the actual
state of a population
monitoring, compliance: monitoring conducted to determine if those who must implement
programs, best management practices, or management measures, or who must conduct
operations according to standards or specifications are doing so
monitoring, project: monitoring conducted to determine the impact of a project, activity, or
program
monitoring, validation: monitoring conducted to determine how well a model accurately
reflects reality
navigational error: errors in determining the actual location (altitude or latitude/longitude) of
an airplane or other aviation device due to instrumentation or the operator
nominal: referred to by name; variables that cannot be measured*but must be expressed
qualitatively
-------
Glossary
nonparametric method: distribution-free method; any of various inferential procedures whose
conclusions do not rely on assumptions about the distribution of the population of interest
normal approximation: an assumption that a population has the characteristics of a normally-
distributed population
normal deviate: deviation from the mean expressed hi units of o
nutrient management plan: a plan for managing the quantity of nutrients applied to crops to
achieve maximum plant nutrition and minimum nutrient waste
ordinal: ordered such that the position of an element in a series is specified
parametric method: any statistical method whose conclusions rely on assumptions about the
distribution of the population of interest
physiography: a description of the surface features of the Earth; a description of land forms
pie chart: a representation of data wherein data is grouped and represented as more or less
triangular sections of a circle and the total is the entire circle
population, sample: the members of a population that are actually sampled or measured
population, target: the population about which inferences are made; the group of interest,
from which samples are taken
population unit: an individual member of a target population that can be measured
independently of other members
power: the probability of correctly rejecting the null hypothesis when the alternative
hypothesis is true
precision: a measure of the similarity of individual measurements of the same population
question, dichotomous: a question that allows for only two responses, such as "yes" and "no"
question, double-barreled: two questions asked as a single question
question, multiple-choice: a question with two or more predetermined responses
question, open-ended: a question format that requires a response beyond "yes" or "no"
-------
remote sensing: methods of obtaining data from a location distant from the object being
measured, such as from an airplane or satellite
resolution: the sharpness of a photograph
sample size: the number of population units measured
sampling, cluster: sampling in which small groups of population units are selected for
sampling and each unit in each selected group is measured
sampling, simple random: sampling in which each unit of the target population has an equal
chance of being selected
sampling, stratified random: sampling in which the target population is divided into separate
subgroups, each of which is more internally similar than the overall population is, prior to
sample selection
sampling, systematic: sampling in which population units are chosen in accordance with a
predetermined sample selection system
sampling error: error attributable to actual variability in population units not accounted for by
the sampling method
scale (aerialphotography): the proportion of the image size of an object (such as a land area)
to its actual size, e.g., 1:3,000; the smaller the second number, the larger the scale
scale system: a system for ranking measurements or members of a population on a scale, such
as 1 to 5
significance level: in hypothesis testing, the probability of rejecting a hypothesis that is
correct, that is, the probability of a Type I error
standard deviation: a measure of spread; the positive square root of the variance
standard error: an estimate of the standard deviation of means that would be expected if a
collection of means based on equal-sized samples of n items from the same population were
obtained
statistical inference: conclusions drawn about a population using statistics
-------
Glossary
statistics, descriptive: measurements of population characteristics designed to summarize
important features of a data set
stratification, the process of dividing a population into internally similar subgroups
stratum: one of the subgroups created prior to sampling in stratified random sampling
streamside management area: a designated area that consists of a water body (e.g., stream)
and an adjacent area of varying width where management practices that might affect water
quality, fish, or other aquatic resources are modified to protect the water body and its adjacent
resources and to reduce the pollution effect of an activity on the water body
subjectivity: a characteristic of analysis that requires personal judgement on the part of the
person doing the analysis
target audience: the population that a monitoring effort is intended to measure
test, analysis of variance: a statistical test used to determine whether two or more sample
means could have been obtained from populations with the same parametric mean
test, Friedman: a nonparametric test that can be used for analysis when two variables are
involved
test, Kniskal-Wallis: a nonparametric test recommended for the general case with a samples
and nt variates per sample
test, Mann-Whitney: a nonparametric test for use when a test is only between two samples
test, student's t: a statistical test used to test for significant differences between means when
only two samples are involved
test, Tukey's: a test to ascertain whether the interaction found hi a given set of data can be
explained in terms of multiplicative main effects
test, Wilcoxon's: a nonparametric test for use when a test is only between two samples
total maximum daily load: a total allowable addition of pollutants from all affecting sources to
an individual water body over a 24-hour period
transformation, data: manipulation of data such that it will meet the assumptions required for
analysis
-------
GloSsiffC!
unit sampling cost: the cost of attributable to sampling a single population unit
variance: a measure of the spread of data around the mean
waterbar: a diversion ditch and/or hump installed across a trail or road to divert runoff from
the surface before the flow gams enough volume and velocity to cause soil movement and
erosion
watershed assessment: an investigation of numerous characteristics of a watershed hi order to
describe its actual condition
-------
-------
APPENDIXA
Statistical Tables
-------
-------
Table Al. Cumulative
toZp)
z
O.I
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
.1
.2
•2
.4
.5
.6
1.7
1.8
1.9
2.0
2.1
2.2
2.3
2.4
2.5
2.6
2.7
2.3
2.9
3.0
3.1
3.2
3.3
3.4
0.00
0.5000
0.5398
0.5793
0.6179
0.6554
0.6915
0.7257
0.7580
0.7881
0.8159
0.8413
0.8643
0.8849
0.9032
0.9192
0.9332
0.9452
0.9554
0.9641
0.9713
0.9772
0.9821
0.9861
0.9893
0.9918
0.9938
0.9953
0.9965
0.9974
0.9981
0.9987
0.9990
0.9993
0.9995
0.9997
^
0.01
0.5040
0.5438
0.5832
0.6217
0.6591
0.6950
0.7291
0.7611
0.7910
0.8186
0.8438
0.8665
0.8869
0.9049
0.9207
0.9345
0.9463
0.9564
0.9649
0.9719
0.9778
0.9826
0.9864
0.9896
0.9920
0.9940
0.9955
0.9966
0.9975
0.9982
0.9987
0.9991
0.9993
0.9995
0.9997
areas under the
A
0.02
0.5080
0.5478
0.5871
0.6255
0.6628
0.6985
0.7324
0.7642
0.7939
0.8212
0.8461
0.8686
0.8888
0.9066
0.9222
0.9357
0.9474
0.9573
0.9656
0.9726
0.9783
0.9830
0.9868
0.9898
0.9922
0.9941
0.9956
0.9967
0.9976
0.9982
0.9987
0.9991
0.9994
0.9995
0.9997
1
0.03
0.5120
0.5517
0.5910
0.6293
0.6664
0.7019
0.7357
0.7673
0.7967
0.8238
0.8485
0.8708
0.8907
0.9082
0.9236
0.9370
0.9484
0.9582
0.9664
0.9732
0.9788
0.9834
0.9871
0.9901
0.9925
0.9943
0.9957
0.9968
0.9977
0.9983
0.9988
0.9991
0.9994
0.9996
0.9997
Normal
\
0.04
0.5160
0.5557
0.5948
0.6331
0.6700
0.7054
0.7389
0.7704
0.7995
0.8264
0.8508
0.8729
0.8925
0.9099
0.9251
0.9382
0.9495
0.9591
0.9671
0.9738
0.9793
0.9838
0.9875
0.9904
0.9927
0.9945
0.9959
0.9969
0.9977
0.9984
0.9988
0.9992
0.9994
0.9996
0.9997
distribution (values of p
0.05
0.5199
0.5596
0.5987
0.6368
0.6736
0.7088
0.7422
0.7734
0.8023
0.8289
0.8531
0.8749
0.8944
0.9115
0.9265
0.9394
0.9505
0.9599
0.9678
0.9744
0.9798
0.9842
0.9878
0.9906
0.9929
0.9946
0.9960
0.9970
0.9978
0.9984
0.9989
0.9992
0.9994
0.9996
0.9997
0.06
0.5239
0.5636
0.6026
0.6406
0.6772
0.7123
0.7454
0.7764
0.8051
0.8315
0.8554
0.8770
0.8962
0.9131
0.9279
0.9406
0.9515
0.9608
0.9686
0.9750
0.9803
0.9846
0.9881
0.9909
0.9931
0.9948
0.9961
0.9971
0.9979
0.9985
0.9989
0.9992
0.9994
0.9996
0.9997
0.07
0.5279
0.5675
0.6064
0.6443
0.6808
0.7157
0.7486
0.7794
0.8078
0.8340
0.8577
0.8790
0.8980
0.9147
0.9292
0.9418
0.9525
0.9616
0.9693
0.9756
0.9808
0.9850
0.9884
0.991 1
0.9932
0.9949
0.9962
0.9972
0.9979
0.9985
0.9989
0.9992
0.9995
0.9996
0.9997
corresponding
0.08
0.5319
0.5714
0.6103
0.6480
0.6844
0.7190
0.7517
0.7823
0.8106
0.8365
0.8599
0.8810
0.8997
0.9162
0.9306
0.9429
0.9535
0.9625
0.9699
0.9761
0.9812
0.9854
0.9887
0.9913
0.9934
0.9951
0.9963
0.9973
0.9980
0.9986
0.9990
0.9993
0.9995
0.9996
0.9997
0.09
0.5359
0.5753
0.6141
0.6517
0.6879
0.7224
0.7549
0.7852
0.8133
0.8389
0.8621
0.8830
0.9015
0.9177
0.9319
0.9441
0.9545
0.9633
0.9706
0.9767
0.9817
0.9857
0.9890
0.9916
0.9936
0.9952
0.9964
0.9974
0.9981
0.9986
0.9990
0.9993
0.9995
0.9997
0.9998
-------
Appendix A
Table A2. Percentiles of the ta^f distribution (values of t such that 100(1-a) % of the
distribution is less than t)
dt
'
2
:
t
6
6
"
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
35
40
50
60
80
100
150
200
inf.
a » 0.40
0.3249
0.2887
0.2767
0.2707
0.2672
0.2648
0.2632
0.2619
0.2610
0.2602
0.2596
0.2590
0.2586
0.2582
0.2579
0.2576
0.2573
0.2571
0.2569
0.2567
0.2566
0.2564
0.2563
0.2562
0.2561
0.2560
0.2559
0.2558
0.2557
0.2556
0.2553
0,2550
0.2547
0.2545
0.2542
0.2540
0.2538
0.2537
0,2533
^
a = 0.30
0.7265
0.6172
0.5844
0.5686
0.5594
0.5534
0.5491
0.5459
0.5435
0.5415
0.5399
0.5386
0.5375
0.5366
0.5357
0.5350
0.5344
0.5338
0.5333
0.5329
0.5325
0.5321
0.5317
0.5314
0.5312
0.5309
0.5306
0.5304
0.5302
0.5300
0.5292
0.5286
0.5278
0.5272
0.5265
0.5261
0.5255
0.5252
0.5244
/~
a = 0.20
* 1.3764
1.0607
0.9785
0.9410
0.9195
0.9057
0.8960
0.8889
0.8834
0.8791
0.8755
0.8726
0.8702
0.8681
0.8662
0.8647
0.8633
0.8620
0.8610
0.8600
0.8591
0.8583
0.8575
0.8569
0.8562
0.8557
0.8551
0.8546
0.8542
0.8538
0.8520
0.8507
0.8489
0.8477
0.8461
0.8452
0.8440
0.8434
0.8416
N
^
a - 0.10
3.0777
1.8856
1.6377
1.5332
1.4759
1.4398
1.4149
1.3968
1.3830
1.3722
1.3634
1.3562
1.3502
1.3450
1.3406
1.3368
1.3334
1.3304
1.3277
1.3253
1.3232
1.3212
1.3195
1.3178
1.3163
1.3150
1.3137
1.3125
1.3114
1.3104
1.3062
1.3031
1.2987
1.2958
1.2922
1.2901
1.2872
1.2858
1.2816
Area
a = 0.05
6.3137
2.9200
2.3534
2.1318
2.0150
1.9432
1.8946
1.8595
1.8331
1.8125
1.7959
1.7823
1.7709
1.7613
1.7531
1.7459
1.7396
1.7341
1.7291
1.7247
1.7207
1.7171
1.7139
1.7109
1.7081
1.7056
1.7033
1.7011
1.6991
1.6973
1.6896
1.6839
1.6759
1.6706
1.6641
1.6602
' 1.6551
1.6525
1.6449
= a
a = 0.025
12.7062
4.3027
3.1824
2.7765
2.5706
2.4469
2.3646
2.3060
2.2622
2.2281
2.2010
2.1788
2.1604
2.1448
2.1315
2.1199
2.1098
2.1009
2.0930
2.0860
2.0796
2.0739
2.0687
2.0639
2.0595
2.0555
2.0518
2.0484
2.0452
2.0423
2.0301
2.0211
2.0086
2.0003
1.9901
1.9840
1.9759
1.9719
1.9600
a = 0.010
31.8210
6.9645
4.5407
3.7469
3.3649
3.1427
2.9979
2.8965
2.8214
2.7638
2.7181
2.6810
2.6503
2.6245
2.6025
2.5835
2.5669
2.5524
2.5395
2.5280
2.5176
2.5083
2.4999
2.4922
2.4851
2.4786
2.4727
2.4671
2.4620
2.4573
2.4377
2.4233
2.4033
2.3901
2.3739
2.3642
2.3515
2.3451
2.3264
a = 0.005
63.6559
9.9250
5.8408
4.6041
4.0321
3.7074
3.4995
3.3554
3.2498
3.1693
3.1058
3.0545
3.0123
2.9768
2.9467
2.9208
2.8982
2.8784
2.8609
2.8453
2.8314
2.8188
2.8073
2.7970
2.7874
2.7787
2.7707
2.7633
2.7564
2.7500
2.7238
2.7045
2.6778
2.6603
2.6387
2.6259
2.6090
2.6006
2.5758
-------
Table A3. Upper and lower percentiles of the Chi-square distribution
df
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
35
40
50
60
70
80
90
100
200
0.001
0.002
0.024
0.091
0.210
0.381
0.599
0.857
1.152
1.479
1.834
2.214
2.617
3.041
3.483
3.942
4.416
4.905
5.407
5.921
6.447
6.983
7.529
8.085
8.649
9.222 '
9.803 '
10.391 '
10.986 '
11.588 '
14.688 '
17.917 ;
24.674 :
31.738 C
39.036 <
46.520 J
54.156 J
61.918 £
143.84 '
f
0.005
0.010
0.072
0.207
0.412
0.676
0.989
1.344
1.735
2.156
2.603
3.074
3.565
4.075
4.601
5.142
5.697
6.265
6.844
7.434
8.034
8.643
9.260
9.886
1 0.520
1.160
1.808
2.461
3.121
3.787
7.192
>0.707
27.991
35.534
13.275
51.172
59.196
57.328
52.24
\
0.010
0.020
0.115
0.297
0.554
0.872
1.239
1.647
2.088
2.558
3.053
3.571
4.107
4.660
5.229
5.812
6.408
7.015
7.633
8.260
8.897
9.542
10.196
10.856
11.524
12.198
12.878
13.565
14.256
14.953
18.509
22.164
29.707
37.485
45.442
53.540
61.754
70.065
156.43
^
X2
0.025
0.001
0.051
0.216
0.484
0.831
1.237
1.690
2.180
2.700
3.247
3.816
4.404
5.009
5.629
6.262
6.908
7.564
8.231
8.907
9.591
10.283
10.982
11.689
12.401
13.120
13.844
14.573
15.308
16.047
16.791
20.569
24.433
32.357
40.482
48.758
57.153
65.647
74.222
162.73
Area
hta__
0.050
0.004
0.103
0.352
0.711
1.145
1.635
2.167
2.733
3.325
3.940
4.575
5.226
5.892
6.571
7.261
7.962
8.672
9.390
10.117
10.851
11.591
12.338
13.091
13.848
14.611
15.379
16.151
16.928
17.708
18.493
22.465
26.509
34.764
43.188
51.739
60.391
69.126
77.929
168.28
= 1-p
P
0.100
0.016
0.211
0.584
1.064
1.610
2.204
2.833
3.490
4.168
4.865
5.578
6.304
7.041
7.790
8.547
9.312
10.085
10.865
11.651
12.443
13.240
14.041
14.848
15.659
16.473
17.292
18.114
18.939
19.768
20.599
24.797
29.051
37.689
46.459
55.329
64.278
73.291
82.358
174.84
0.900
2.706
4.605
6.251
7.779
9.236
10.645
12.017
13.362
14.684
15.987
17.275
18.549
19.812
21.064
22.307
23.542
24.769
25.989
27.204
28.412
29.615
30.813
32.007
33.196
34.382
35.563
36.741
37.916
39.087
40.256
46.059
51.805
63.167
74.397
85.527
96.578
107.57
118.50
226.02
0.950
3.841
5.991
7.815
9.488
11.070
12.592
14.067
15.507
16.919
18.307
19.675
21.026
22.362
23.685
24.996
26.296
27.587
28.869
30.144
31.410
32.671
33.924
35.172
36.415
37.652
38.885
40.113
41.337
42.557
43.773
49.802
55.758
67.505
79.082
90.531
101.88,
113.15
124.34
233.99
0.975
5.024
7.378
9.348
11.143
12.832
14.449
16.013
17.535
19.023
20.483
21.920
23.337
24.736
26.119
27.488
28.845
30.191
31.526
32.852
34.170
35.479
36.781
38.076
39.364
40.646
41.923
43.195
44.461
45.722
46.979
53.203
59.342
71.420
83.298
95.023
106.63
118.14
129.56
241.06
0.990
6.635
9.210
11.345
13.277
15.086
16.812
18.475
20.090
21.666
23.209
24.725
26.217
27.688
29.141
30.578
32.000
33.409
34.805
36.191
37.566
38.932
40.289
41.638
42.980
44.314
45.642
46.963
48.278
49.588
50.892
57.342
63.691
76.154
88.379
100.43
112.33
124.12
135.81
249.45
0.995
7.879
10.597
12.838
14.860
16.750
18.548
20.278
21.955
23.589
25.188
26.757
28.300
29.819
31.319
32.801
34.267
35.718
37.156
38.582
39.997
41.401
42.796
44.181
45.558
46.928
48.290
49.645
50.994
52.335
53.672
60.275
66.766
79.490
91.952
104.21
116.32
128.30
140.17
255.26
0.999
10.827
13^815
16.266
18.466
20.515
22.457
24.321
26.124
27.877
29.588
31.264
32.909
34.527
36.124
37.698
39.252
40.791
42.312
43.819
45.314
46.796
48.268
49.728
51.179
52.619
54.051
55.475
56.892
58.301
59.702
66.619
73.403
86.660
99.608
112.32
124.84
137.21
149.45
267.54
-------
-------
APPENDIX B
Sample Evaluation Forms
U.S. Forest Service, Region 5 B-l
Minnesota Department of Natural Resources, Division of Forestry B-2, B-3
Texas Forest Service, Forest Resource Development Department B-4
-------
-------
-------
Appendix B
SITE NUMBER:
OWNERSHIP:
LEGAL DESCRIPTION:
PROJECT ACRES REVIEWED:
DATE:
OPERATOR:
SALE OR PROJECT NUMBER:
TEAM INITIALS:
SITE CONDITIONS
PRACTICES
LANDFORM:
GENERAL SOILS:
DRAINAGE:
SLOPE RANGE:
WATER BODIES PRESENT (type):
DEPTH/WIDTH OF STREAMS(type):
OTHER:
STAGE ("x" if completed)
PREHARVEST( ) ROAD CONSTRUCTION ( )
HARVEST ( ) SLASH DISPOSAL ( )
SITE PREP ( )
DATE OF ACTIVITY
ROADS:
NEW CONSTRUCTION (length):
RECONSTRUCTION (length):
HARVEST ACRES:
HARVEST METHOD:
SITE PREP ACRES:
SITE PREP METHOD:
SLASH DISPOSAL:
PESTICIDES USED:
OTHER:
RATING GUIDE
APPLICATION
5-OPERATION EXCEEDS REQUIREMENT OF BMP
4-OPERATION MEETS REQUIREMENT OF BMP
3-MINOR DEPARTURE FROM BMP
2-MAJOR DEPARTURE FROM BMP
1-GROSS NEGLECT OF BMP
EFFECTIVENESS
6-IMPROVED PROTECTION OF SOIL AND WATER RESOURCES OVER PRE-PROJECT CONDITION.
5-ADEQUATE PROTECTION OF SOIL AND WATER RESOURCES.
4-MINOR AND TEMPORARY IMPACTS ON SOIL AND WATER RESOURCES.
3-MAJOR AND TEMPORARY IMPACTS ON SOIL AND WATER RESOURCES.
2-MINOR AND PROLONGED IMPACTS ON SOIL AND WATER RESOURCES.
1-MAJOR AND PROLONGED IMPACTS ON SOIL AND WATER RESOURCES.
DEFINITIONS (BY EXAMPLE)
ADEQUATE: Small amount of material eroded; material does not reach drainages, streams, lakes or wetlands
MINOR: Erosion and delivery of material to drainages but not to streams, lakes or open-water wetlands.
MAJOR: Erosion and subsequent delivery of sediment to streams, lakes or open water wetlands.
TEMPORARY: Impacts lasting one year or less; no more than one runoff season.
PROLONGED: Impacts lasting more than one year.
* It Is possible to have a departure from BMPs and still adequate protection.
-------
RECOMMENDED BEST MANAGEMENT APPLICABLE TO SITE (Y/N) COMMENTS
PRACTICES APPLICATION
EFFECTIVENESS
MECHANICAL SITE PREP
15 General Recommendations (pso>
15a Site prep technique appropriate to the site
15b Provide adequate filter strips
15c Avoid operating during periods of saturated soil
15d Maintain adequate vegetation adjacent to
designated trout streams
15e Site prep technique properly employed (p50-52)
- Shearing and raking
- Disking
- Patch or row scarification
-Other
PESTICIDE USE
16 Prevent entry of pesticide residues into surface
and ground waters (p57-75)
PRESCRIBED BURNING
17 Planning
-------
Appendix B
GENERAL
1. County
3. Latitude
Forester: 4.
6. Timber Buyer.
7. Logger
TEXAS BMP MONITORING CHECKLIST
SITE ID No:
_2. Block/Grid.
_ Longitude
5.
8. Activity
9. Estimated date of activity _
10. Acres affected
11. Inspector
LANDOWNER:
12. Owner Type:
13. Name
N L A I P
14. Address
15. City
ZIP
16. Phone
17. Date of Inspection
18. Accompanied by:
SITE CHARACTERISTICS
19. Terrain: F H s
20. Erodability hazard: L M H
21. Type stream present p I
22. Distance to nearest permanent water body:
<300' 300-800' 800-1600' 1600' +
23. Predominant soil series/texture: / c CL L SL s
PERMANENT ROADS
[ ] NOT APPLICABLE
24. Avoid sensitive areas. Y N NA
25. Roads meet grade specs. Y N NA
26. Stabilized stream crossing. Y N NA
27. Rutting within allowable specs. Y N .NA
28. Ditches do not dump into streams. Y N NA
29. Were BMP's used. Y N NA
Type: RD WD WB RE oc PL RS cu BR LW
30. Were BMP's effective. Y N NA
31. Stream free of sediment. Y N NA
SKID TRAILS / TEMPORARY ROADS
[ ] NOT APPLICABLE
32. Slopes less than 15%.
33. Rutting within allowable specs.
34. Water bars evident.
35. Water bars working.
36. Stream crossings mmimimA
37. Stream crossings correct.
38. Stream crossings restored & stabilized.
39. Were BMP's used.
Type: RD WD WB RE oc PL RS cu BR LW
40. Stream free of sediment.
Y N NA
Y N NA
Y N NA
Y N NA
Y N NA
Y N NA
Y N NA
Y N NA
Y N NA
SMZ
[ ] NOT APPLICABLE
41. SMZ present on permanent stream. Y N NA
42. SMZ present on intermittent stream. Y N NA
43. SMZ adequately wide. Y N NA
44. Thinning within allowable specs. Y N NA
45. SMZ integrity honored.
46. Stream clear of debris.
47. SMZ free of roads and landings.
48. Stream free of sediment.
Y N NA
Y N NA
Y N NA
Y N NA
SITE PREPARATION
[ ] NOT APPLICABLE
49. Site prep method
50. Regeneration method
51. No soil movement on site.
52. Firebreak erosion controlled.
53. SMZ integrity honored.
54. Windrows on contour / free of soil. Y N NA
55. No chemicals off site. Y N NA
Y N NA 56. Were BMP's used. Y N NA
Y N NA Type: WB RE oc RS
Y N NA 57. Stream free of sediment. Y N NA
LANDINGS
[ ] NOT APPLICABLE
58. Locations free of oil / trash.
59. Located outside SMZ.
Y N NA 60. Well drained location
Y N NA 61. Restored, stabilized.
Y N
Y N NN
62. Overall compliance with Best Management Practices
Set Eviiuilion Crtlerii for • full description or numbered questions.
NEEDS IMPROVEMENT
NO EFFORT POOR
PASS
FAIR GOOD EXCELLENT
-------
Index
Aerial Photography, 4-17
Best Management Practice Evaluation
Program, U.S. Forest Service, 2-15
Bias
measurement, 2-8
BMP
binary rating system, 4-9, 4-10
compliance, 1-4
design and installation, 1-4
effectiveness, 1-1, 1-4, 2-1
implementation, 1-1, 1-3
implementation plans, 1-3
rating, 4-9
rating terms, 4-11
scale rating system, 4-10
standards and specifications, 1-4
voluntary implementation, 1-3
BMP implementation
and water quality, 1-1
Bureau of Land Management, 2-14, 4-18
CNPCP, 1-2, 1-3
Company X, 2-21, 2-23, 2-24
Confidence interval, 2-11
Contingency table, 3-4
Cumulative effects
of BMP implementation, 1-3
CWA, 1-2
303(d), 1-2
319(h), 1-2
CZARA, 1-1
6217(b), 1-2, 1-2
6217(d), 1-2, 1-2
6217(g), 1-2
technical assistance, 1-2, 1-2
Data
categorical, 3-4
evaluation, 3-1
nominal, 3-4
ordinal, 3-4, 3-6
EPA, 1-2, 1-2, 2-1, 2-19
Error
allowable, 2-19
measurement, 2-8, 2-10
relative, 2-19
sampling, 2-8
Type I, 2-12
Type H, 2-12
Estimation, 2-10
Evaluation
variable selection, 4-4
variables, 4-2
Evaluations
consistency, 4-8
expert, 4-4
presentation of results, 5-1
self, 4-1, 4-13
site, 4-4
teams, 4-8
Guidance Specifying Management Measures
For Sources Of Nonpoint Pollution in
Coastal Waters, 1-2
Hypothesis
testing, 2-1
Hypothesis testing, 2-10
Kendall tb, 3-6
Management
goals and objectives, 1-4
Monitoring
baseline, 1-4, 1-4
compliance, 1-4
Effectiveness, 1-4, 1-4, 2-1
goals, 2-1
implementation, 1-1, 1-3, 2-1
long-term, 1-3
objectives, 2-1
project, 1-4
purpose, 5-1
purpose of, 1-4
scale, 1-1, 1-3
trend, 1-3, 1-4
-------
types, 1-3
validation, 1-4
water quality, 1-1
Monitoring programs
Florida, 2-14
Idaho, 2-15
Montana, 2-15
US Forest Service, 2-15
NOAA, 1-2, 1-2
Nonpoint Source Monitoring and Evaluation
Guide, 1-5
NPDES, 1-2
Photography
aerial, 2-14, 4-17
altitude, 4-19
drawback, 4-19
large-format, 4-18
navigational errors, 4-19
resolution, 4-18
scale, 4-18
small-format, 4-19
training, 4-19
Population
periodic variation, 2-8
sample, 2-2
target, 2-2, 2-2
units, 2-2
Presentations
audience, 5-2
criteria, 5-1
format, 5-2, 5-2
graphics, 5-3
medium, 5-2
oral, 5-3
potential audiences, 5-2
references, 5-6
use of VCR, 5-6
Written, 5-3
Questionnaire, 4-1
design, 4-15
dichotomous questions, 4-17
double-barreled questions, 4-16
layout, 4-17
multiple-choice questions, 4-17
open-ended questions, 4-17
ordering of questions, 4-17
phrasing of questions, 4-16
pretest, 4-17
refusal to answer, 4-16
response format, 4-17
revision, 4-17
target audience ,4-16
Sample size, 2-17
Sampling, 2-1
accuracy, 2-10
alternative hypothesis, 2-12
balanced designs, 2-2
bias, 2-10, 2-14
cluster, 2-5, 2-26
comparing more than two, 3-4
comparison of means, 3-2
comparison of proportions, 3-3
cost-effectiveness, 2-5
distribution, 3-3
intervals, 2-8
Neyman allocation, 2-24
null hypothesis, 2-11
power, 2-12
precision, 2-10
Probabilistic, 2-2
program design, 2-1
proportional allocation, 2-24
simple random, 2-2, 2-19
site selection, 2-13
site selection criteria, 2-13
site selection data, 2-14
sources of variability, 2-10
strategy, 2-13
stratified random, 2-3, 2-23
systematic, 2-3, 2-8, 2-26
': water quality, 2-1
Significance level, 2-12, 2-17
-------
Site-specific, 1-3
Statistical inference, 2-2
Statistics
descriptive, 2-10
estimation, 2-17
finite population correction term,
2-17
summary, 2-2
Surveys
accuracy of information, 4-14
Cost, 4-14
landowner involvement, 4-13
mail, 4-14, 4-17
postevaluation activities, 4-13
site visit, 4-14
telephone, 4-14
Test
analysis of variance, 3-4
Friedman, 3-4
Kruskal-Wallis, 3-4, 3-5
Mann-Whitney, 3-2, 3-2
one-sided, 3-1
Student's t, 2-20, 3-2
Tukey's, 3-4
two-sided, 3-1
Wilcoxon's, 3-2
US Forest Service, 2-16
Variables
examples of good, 4-2
Watershed assessment, 1-3
-------
------- |