EPA-670/2-74-082
November 1974
Environmental Protection Technology Series
MEASURES OF EFFECTIVENESS FOR
REFUSE STORAGE, COLLECTION, AND
TRANSPORTATION PRACTICES
National Environmental Research Center
Office of Research and Development
U.S. Environmental Protection Agency
-------
EPA-670/2-74-082
November 1974
MEASURES OF EFFECTIVENESS FOR REFUSE STORAGE,
COLLECTION, AND TRANSPORTATION PRACTICES
By
MESSER ASSOCIATES, INC.
Silver Spring, Maryland 20910
Program Element No. 1DB063
Project Officer
Albert J. Klee
Solid and Hazardous Waste Research Laboratory
National Environmental Research Center
Cincinnati, Ohio 45268
NATIONAL ENVIRONMENTAL RESEARCH CENTER
OFFICE OF RESEARCH AND DEVELOPMENT
U.S. ENVIRONMENTAL PROTECTION AGENCY
CINCINNATI, OHIO 45268
-------
REVIEW NOTICE
The National Environmental Research Center--
Cincinnati has reviewed this report and approved
its publication. Approval does not signify that
the contents necessarily reflect the views and
policies of the U.S. Environmental Protection
Agency, nor does mention of trade names or com-
mercial products constitute endorsement or recom-
mendation for use.
-------
FOREWORD
Man and his environment must be protected from the
adverse effects of pesticides, radiation, noise and other
forms of pollution, and the unwise management of solid
waste. Efforts to protect the environment require a focus
that recognizes the interplay between the components of our
physical environment—air, water, and land. The National
Environmental Research Centers provide this multidisciplinary
focus through programs engaged in
• studies on the effects of environmental contaminants
on man and the biosphere, and
• a search for ways to prevent contamination and to
recycle valuable resources.
This report presents results of a project for that focussed
on the systematic development of a set of measures and measurement
tools that could be used to assess the effectiveness of solid
waste storage, collection, and transportation practices. The
measurement system presented is intended to support municipal
decision-makers who have responsibility for such services as
mixed refuse collection, street and alley cleaning, sanitary
code enforcement, sanitation education, and other related
activities.
Andrew W. Breidenbach, Ph.D.
Director
National Environmental
Research Center, Cincinnati
-------
ABSTRACT
Perhaps between 75 to 80 percent of a solid waste system
cost is due to storage, collection, and transportation, the
remainder being attributable to disposal. Given an adequate
accounting system, the monetary costs of a solid waste manage-
ment system are much easier to compute than are the benefits
produced and the nonmonetary cost incurred. Thus, although
a community may have an accurate estimate of what it is spending
upon its system, it often is uncertain as to whether or not it is
receiving reasonable value in benefits returned; i. e., it has
little or no idea of its "cost effectiveness."
This report presents the results of a project that focussed
on the systematic development of a set of measures and measure-
ment tools that could be used to assess the effectiveness of solid
waste storage, collection, and transportation practices. The
project included a pilot test of the measurement methodology in
an urban community.
The measurement system presented in this report is
intended to support municipal decision-makers who have re-
sponsibility for such services as mixed refuse collection, street
and alley cleaning, sanitary code enforcement, sanitation educa-
tion, and other related activities. It provides a model or proto-
type that municipal representatives can use to design effectiveness
measures that are specific to their own solid waste management
needs and activities.
The report includes a comprehensive list of candidate
effectiveness measures along with the measurement techniques
and sampling procedures needed to collect data to formulate the
candidate measures. It also includes methods for combining
individual measures into overall effectiveness indices.
This report was submitted in fulfillment of Contract
Number 68-03-0260 by Messer Associates, Inc. under the
sponsorship of the Environmental Protection Agency. Work was
completed as of June 1974.
IV
-------
TABLE OF CONTENTS
Page
Number
Abstract
List of Figures
List of Tables
Acknowledgments
Summary of Findings and Recommendations
IV
vii
ix
xiii
xiv
I. Introduction
II. Background Materials Review
III. Development of Measures and Measurement
Techniques
19
IV. Development of Analytical Methods for Combining
Component Variable Measurements to Produce an
Overall Effectiveness Measure
37
V. The Field Demonstration
49
VI. Findings and Conclusions
63
VII. Recommendations
101
-------
Page
Number
Cited References 105
Appendix A: Further Discussion of the Linear,
Conjunctive, and Disjunctive Decision
Models 107
Appendix B: An Assessment of the Usefulness of the
Candidate Effectiveness Measures 113
Appendix C: Description of the Survey Design and its
Implementation During the Pilot Test 121
Appendix D: The Data Collection Forms and Procedures 129
Appendix E: Description of the Analysis Plan 147
Appendix F: Tables, Charts, and Graphs that Support
Findings 161
VI
-------
LIST OF FIGURES
Page
Number
1. "Measure-Act-Measure" Mechanism for Evaluating
Solid Waste Activities 4
2. Overview of the General Framework for Developing
Candidate Effectiveness Measures 23
3. Replica of a Block Map That Was Prepared for the
Field Test 55
4. A Sketch of the Six Observational Areas That Were
Inspected in Each Survey Block 59
5. Mean Garbage Rating Difference by Scale Point 75
6. Mean Glass Rating Difference by Scale Point 76
7. Mean Refuse Rating Difference by Scale Point 77
8. Blockface Glass and Refuse Ratings of Observers
Compared with Overall Ratings for Blocks in
Common Tract 81
9. Graphical Presentation of the Regression Equations
Developed for Blockface Data: Relationship Between
Refuse Rating and Overall Effectiveness Rating When
Glass Rating is Held Constant 90
10. Graphical Presentation of the Regression Equations
Developed for Blockface Data: Relationship Between
Glass Rating and Overall Effectiveness Rating When
Refuse Rating is Held Constant 91
A-l Geometric Representations of the Conjunctive Model 109
A-2 Geometric Representations of the Disjunctive Model 111
VII
-------
Page
Number
C-l Geographical Distribution of Baltimore City Census
Tracts Included in the Field Test 127
D-l Replica of the Pre-Survey Form 130
D-2 Replica of the Blockfaces, Alleys, and Private Ways
Data Form 133
D-3 Replica of the Storage and Backyard Area Data Form 138
D~4 Replica of the Vacant Lots, Public Parks, and
Parking Lots Data Form 143
D-5 Replica of the Summary Form 145
F-l Frequency Distribution for Garbage, Glass, and
Refuse Rating Scales — All Tracts in Sample 175
F-2 Frequency Distribution for Garbage, Glass, and
Refuse Rating Scales —Common Tract Only 176
F-3 Average Garbage Rating for Blockfaces in the Common
Tract: Overall and by Rater 177
F-4 Average Glass Rating for Blockfaces in the Common
Tract: Overall and by Rater 178
F-5 Average Refuse Rating for Blockfaces in the Common
Tract: Overall and by Rater 179
F-6 Average Garbage Rating for Alleys in the Common
Tract: Overall and by Rater 180
F-7 Average Glass Rating for Alleys in the Common Tract:
Overall and by Rater 181
F-8 Average Refuse Rating for Alleys in the Common Tract:
Overall and by Rater 182
Vll 1
-------
LIST OF TABLES
Page
Number
1. Summary of Major Categories for Measuring
Effectiveness 26
2. Indicators of Effectiveness for Each Measurement
Category, Matched to Solid Waste System Operations 28
3. Candidate Effectiveness Measures and Measurement
Techniques by Measurement Category, Indicator,
and Activity 30
4. Summary of the Information Collected and the
Recording Procedures by Type of Measurement 60
5. Distributional Characteristics of the Variables 65
6. Blockface Mean Garbage Ratings for Census Tracts —
Ratings Based on All Blockfaces in a Block Versus
Ratings Based on One Randomly Selected Blockface
Per Block 69
7. Blockface Mean Glass Ratings for Census Tracts —
Ratings Based on All Blockfaces in a Block Versus
Ratings Based on One Randomly Selected Blockface
Per Block 70
8. Blockface Mean Refuse Ratings for Census Tracts —
Ratings Based on All Blockfaces in a Block Versus
Ratings Based on One Randomly Selected Blockface
Per Block 71
9. Rater Agreement for Measurements Other Than
Rating Scales 72
10. Rater Agreement for Rating Scale Measurements 73
ix
-------
Page
Number.
11. The Variance Components for a Census Tract Mean
Value for Glass and Refuse Ratings in Blockfaces and
Alleys 79
12. Differences Between the Average Rater Values and the
Overall Average Values of the Garbage, Glass, and
Refuse Ratings for Blockfaces and Alleys in the Common
Tract 80
13. Correlations Among the Variables Used to Measure
Blockface Conditions &1
14. Correlations Among the Variables Used to Measure
Alley Conditions 87
15. Index Formulas for Blockface Conditions 93
16. Index Formulas for Alley Conditions 95
17. Average Refuse, Glass, and Garbage Ratings for
Blockfaces and Alleys by Stratum 96
18. Amount of Change in the Census Tract Mean Blockface
Refuse Ratings Detectable at the 95 Percent Confidence
Level 98
19. Amount of Change in the Census Tract Mean Alley
Refuse Ratings Detectable at the 95 Percent Confidence
Level 99
B-l Variable Scores for the Candidate Effectiveness
Measures by Solid Waste Activity and by Measure-
ment Category 114
B-2 Variable Scores for Each of the Candidate Effective-
ness Measures by Measurement Category 116
C-l Basic Survey Design 124
C-2 Distribution of Field Test Census Tracts by Income
Grouping and Sanitation District 128
-------
Page
Number
C-3 Distribution of Baltimore City Census Tracts by
Income Grouping and Sanitation District 128
F-l Frequency Distribution for Bulk Items by Area of
Observation — All Tracts in Sample 162
F-2 Frequency Distribution for Bulk Items by Area of
Observation — Common Tract Only 162
F-3 Frequency Distribution for Dead Animals by Area of
Observation — All Tracts in Sample 163
F-4 Frequency Distribution for Dead Animals by Area of
Observation — Common Tract Only 163
F-5 Frequency Distribution for Abandoned Vehicles by
Area of Observation — All Tracts in Sample 164
F-6 Frequency Distribution for Abandoned Vehicles by
Area of Observation — Common Tract Only 164
F-7 Frequency Distribution for Clogged Drain Basins by
Area of Observation — All Tracts in Sample 165
F-8 Frequency Distribution for Clogged Drain Basins by
Area of Observation — Common Tract Only 165
F-9 Frequency Distribution for Fire Hazards by Area of
Observation — All Tracts in Sample 166
F-10 Frequency Distribution for Fire Hazards by Area of
Observation — Common Tract Only 166
F-ll Frequency Distribution for Rat Indicators by Area of
Observation — All Tracts in Sample 167
F-l2 Frequency Distribution for Rat Indicators by Area of
Observation — Common Tract Only 167
F-13 Frequency Distribution for Insect Indicators by Area of
Observation — All Tracts in Sample 168
XI
-------
Page
Numberi
F-14 Frequency Distribution for Odors by Area of Obser-
vation — All Tracts in Sample 168
F-15 Garbage Rating Summary Statistics for Consistency
Among Raters 169
F-16 Glass Rating Summary Statistics for Consistency
Among Raters
F-17 Refuse Rating Summary Statistics for Consistency
Among Raters
F-18 Blockface Estimating Equations for Linear, Conjunctive,
and Disjunctive Models for Observer 1 172
F-19 Blockface Estimating Equations for Linear, Conjunctive,
and Disjunctive Models for Observer 2 172
F-20 Blockface Estimating Equations for Linear, Conjunctive,
and Disjunctive Models for Observer 6 173
F-21 Blockface Estimating Equations for Linear, Conjunctive,
and Disjunctive Models for Observer 7 173
F-22 Blockface Estimating Equations for Linear, Conjunctive,
and Disjunctive Models for Observer 10 174
XII
-------
ACKNOWLEDGMENTS
The project team wishes to extend its appreciation to the
Project Officer, Dr. Albert J. Klee of the National Environ-
mental Research Center, for his guidance and assistance during
the conduct of this project. Additionally, we wish to express
our gratitude to the following municipal government personnel
in the City of Baltimore for their help and cooperation during
the pilot demonstration phase of the project.
Dr. Pierce Linaweaver, Director of Public Works,
Department of Public Works
Mr. R.G. Deitrich, Director of Technical Services,
Department of Public Works
Mr. George Schucker, Assistant Commissioner of
Health, Director of Sanitary Services, Department
of Health
Mr. C. Edward Sachs, Director of Bureau of Environ-
mental Hygiene, Department of Health
Mr. Charles A. Carroll, Chief of Rat Eradication
Program, Department of Health
Mr. Neil Curran, Department of Planning
Mr. G.L. Neff, Head of Bureau of Utility Operations,
Department of Public Works
Mr. Edward J. Moore, Chief of Division of Sanitation,
Department of Public Works
Mr. George Winfield, Technical Services, Department
of Public Works
Mr. Reuben Dagold, Bureau of Environmental Hygiene,
Department of Health
Mr. Mark Forester, Department of Planning
Xlll
-------
SUMMARY OF FINDINGS AND RECOMMENDATIONS
This report presents the results of a project that focussed
on the systematic development of a set of measures and measure-
ment tools that could be used by solid waste management agencies
to evaluate the effectiveness of their solid waste operations of
storage, collection, and local transportation.
The objectives of the project were to develop:
• Usable measures for assessing the effectiveness of
the solid waste operations of storage, collection, and
local transportation.
• Measurement techniques and sampling procedures to
obtain data to formulate the measures.
• Methods for combining measurements of individual
variables to yield overall effectiveness indices.
It was not the intent of this project to develop a rigid set of
measures to be used by all agencies. Rather, the intent of the
project was to develop a measurement system that could serve
as a guide (or prototype) for various communities in the develop-
ment of indicators that are specific to their own solid waste man-
agement needs and activities. In addition to providing a model
that local communities could adapt and/or tailor to their needs,
the project was intended to provide a mechanism that could be
used by state and federal solid waste management agencies for
comparing the effectiveness of the solid waste activities performed
by jurisdictions under their control.
To ensure that the measures and the accompanying scheme
for collecting the requisite data would be of use to solid waste
managers, the project was to include a field test of the measure-
ment methodology. This test was to take place in an urban com-
munity.
xiv
-------
MAJOR FINDINGS OF THE PROJECT
The findings and conclusions that were developed on the
basis of the data collected during the field demonstration phase
of the project were of two types: findings of a general nature and
findings related to the sample design.
The findings which are general in nature may be summa-
rized as follows:
The number of variables measured can be reduced
because many of the variables were frequently found
to be at their lowest value.
Only three observational areas need to be inspect-
ed—blockfaces, alleys, and lots.
For blockface measurements, one blockface selected
at random can be used in lieu of all four blockfaces
to measure the effectiveness variables.
Observers exhibit a high degree of consistency for
the "yes-no" type measurements and for "counts. "
Observers exhibit a fair degree of consistency for the
more subjective-type measurements; i. e., glass,
garbage, and refuse ratings. The amount of varia-
tion is less at lower scale points than at higher scale
points, tending to rapidly increase and then stabilize.
Variation among observers, however, only accounts
for approximately 15 to 20 percent of total variation
when measuring a tract mean.
On the whole, observers tend to be accurate to within
one-half of a scale point for the glass, garbage, and
refuse ratings; a few, however, were always high or
always low in their assigned ratings.
There are a number of statistically significant corre-
lations among the variables; however, the explained
variation tends to be low.
xv
-------
It is possible to develop composite measures of effec-
tiveness, using multivariate techniques; however, the
refuse rating by itself can serve as a proxy for an
overall measure.
In summary, these findings indicate that an ongoing measurement
system would require collection of data on only a few variables;
that subjective measurements of unsightly conditions, health
hazards, safety hazards, and so forth can be made, provided that
one is willing to accept a small amount of inconsistency in the
measurements; and that formulation of composite effectiveness
indices is possible, but perhaps unnecessary.
In addition to the general findings described above, there
were several findings that related to the sample design. They
may be summarized as follows:
Large differences in the mean tract ratings were
found to exist between two groupings of the strata.
The sampling plan was adequate to detect changes
of one-half a point or less in the blockface ratings;
larger samples would be required to detect an equiv-
alent change in the alley ratings.
The mean tract ratings did not appear to be biased
by the day of the week when the inspection was made.
MAJOR RECOMMENDATIONS OF THE PROJECT
The recommendations stemming from this project are of
two types:
General recommendations on how to develop a mea-
surement system that is specific to a given commu-
nity.
Detailed recommendations on how to implement an
ongoing measurement system using the findings from
the field test data.
xvi
-------
The recommendations on how to develop a measurement
system that is specific to a given community are as follows:
(1) Review the list of measures and measurement tech-
niques provided in Chapter III of the report and select
those measures most useful.
(2) Use the basic survey design developed for this project
to obtain preliminary data on those measures that re-
quire direct observation of existing conditions.
(3) Utilize several observers in each tract and have them
make the same measurements.
(4) Apply techniques similar to those used in this project
to determine the appropriate sample size for an on-
going measurement system and the relevant variables
for which measurements should be made.
The recommendations on how to implement an ongoing mea-
surement system that draws upon results of the field demonstration
conducted in the City of Baltimore are as follows:
(1) Collect data on blockfaces, alleys, and lots only.
(2) Sample approximately ten blocks in each census tract
where the overall conditions are bad to detect a change
of one-half a point or less in the blockface garbage,
glass, and refuse ratings; inspect fewer blocks in
areas where the overall conditions are good. Use the
income level of the census tract as an initial means
for classifying conditions.
(3) Inspect only one randomly selected blockface in each
block; inspect all alleys and lots in the block.
(4) Utilize several observers and have them inspect dif-
ferent blocks in the same census tract in order to
reduce the variation associated with inconsistency
among observers.
xvi i
-------
(5) Periodically compare the observations of raters
within the same tract to see if any of the observers
are consistently high or consistently low.
(6) Make measurements only of the amount of refuse
found in these areas. Use this as an indicator of
overall conditions.
or
Make measurements of the amount of refuse, glass,
and garbage found in the areas, the presence of rat
signs (alleys only) and the number of bulk items
(alleys only). Report the measurements separately
and/or as a composite measure.
(7) Report the location of fire hazards, bulk items,
abandoned vehicles, clogged basins, and other items
of interest so that corrective action can be taken.
XVlll
-------
I. INTRODUCTION
In many communities, information is available to determine
the costs of solid waste services; i. e., the labor and equipment
costs associated with collection of mixed refuse, street sweeping
and cleaning, and refuse disposal. Similarly, information is gen-
erally available to assess the operational efficiency of solid waste
operations, such as tons of refuse collected per man-hour, miles
of streets cleaned per man-hour ^ and so forth. However, informa-
tion by which to evaluate the effectiveness of sanitation activities
is typically lacking; i. e., information on the degree to which
streets and alleys are kept free from debris so as to prevent con-
ditions harmful to the health and safety of the public and to promote
an aesthetically pleasing environment.
Opinions on sanitation conditions are usually reported by
sanitation crew supervisors. Unfortunately, this is usually a
sporadic and subjective process, characterized by the lack of
well-defined measures that have been agreed upon, and the absence
of a structured mechanism to relate this information to data on
costs and operating activities. The need to clearly delineate the
many variables that are affected by solid waste operations (par-
ticularly storage, collection, and transportation activities) and
develop usable means of quantifying and measuring changes in
these variables is the primary reason why this project was under-
taken.
This chapter describes the objectives of the project and the
potential management uses of effectiveness indicators. It also
provides an overview of how we performed the project and how
this report is organized.
OBJECTIVES OF THE PROJECT
The primary purposes of this project were to develop:
Usable measures to assess the effectiveness of solid
waste operations, with emphasis on the functions of
storage, collection, and local transportation.
-------
Measurement techniques and sampling procedures to
obtain data to formulate the measures.
Methods for combining measurements of individual
variables to yield overall effectiveness measures.
The basic thrust of the project was on the systematic de-
velopment of a set of indicators and appropriate measurement
tools that could be used by solid waste management agencies to
evaluate the effectiveness of their solid waste operations.
It was not the intent of this project to develop a rigid set of
measures to be used by all agencies. Rather, the intent of the
project was to develop a measurement system that could serve
as a guide (or prototype) for various communities in the develop-
ment of indicators that are specific to their own solid waste man-
agement needs and activities. This is a particularly important
consideration because each different solid waste management
agency is likely to have its own ideas on what constitutes meaning-
ful effectiveness indicators. The operating characteristics of the
local agency and the environmental characteristics of the community
that it serves may require a unique set of indicators. Furthermore,
the adequacy of current information systems and the corresponding
ability of individual agencies to generate measures may vary widely
across locales.
In addition to providing a model that local communities could
adapt and/or tailor to their needs, the project was intended to pro-
vide a mechanism that could be used by state and federal solid
waste management agencies for comparing the effectiveness of
the solid waste activities performed by jurisdictions under their
control. By adopting a standardized reporting system that re-
quires a sufficient degree of conformity in the types of data that
are collected and in the data collection procedures, it would be
possible for these agencies to compare the effectiveness of local
solid waste management activities. This, in turn, would provide
state and federal agencies with information to facilitate their
planning, financing, and regulatory responsibilities.
To ensure that the measures and the accompanying scheme
for collecting the requisite data would be of use to solid waste
managers, the project was to include a field test of the measure-
ment methodology. This test was to take place in an urban com-
munity.
-------
MANAGEMENT USES OF EFFECTIVENESS MEASURES
The measures developed during this project provide solid
waste managers with quantitatively based feedback information
on how well the goals and objectives of their solid waste activities
are being met, particularly storage, collection, and local trans-
portation activities. The measures indicate whether changes may
be needed in the underlying policies that govern a given mode of
operation (e.g., frequency of collection, type of collection service,
level of sanitary enforcement activities, etc.) or in the level and
mix of resources devoted to these activities. Where a change is
undertaken, they provide a means for assessing whether this change
has produced the desired impact.
Effectiveness measures are particularly useful in:
Determining those service areas most in need of
corrective action.
Assessing the impact of changes in policy and in the
allocation of resources among various service areas.
Assessing the impact of special sanitation operations
(e. g., anti-litter campaigns, special cleanup cam-
paigns, etc. ).
Preparing annual budgets and justifying budgetary
appropriations for solid waste activities, particularly
when additional resources are needed.
Establishing standards for evaluating the performance
of private waste collectors.
Justifying to citizen groups the type and level of ser-
vice that their area is receiving.
However, as illustrated in Figure 1 on the following page,
effectiveness indicators are only one of several ways that managers
have of evaluating solid waste activities. Other indicators include
performance or output measures (tons of refuse collected per week,
number of streets cleaned per day), and productivity or efficiency
measures (tons of refuse collected per man-hour). These three
-------
POLICY MEASURES
— Storage
• Type of
Containers
* Availability of
Receptacles
• Enforcement of
Standards
• Use of Public
Educ. Campaigns
— Collection
• Frequency of
Regular Collection
• Frequency of Street
and Alley Cleaning
• Frequency of
Special Pickup
• Type of Collection
Equipment
• Route Design
and Crew Size
• Route Schedule
(a.m., p.m.)
• Type of Collection
Service
— Local Transportation
• Type of Transport
Equipment
• Distance
Travelled
• Tons Carried
per Trip
Application of
Resources
Measurement of System
Operations
• Output Measures
• Productivity Measures
• Effectiveness Measures
Review of Results
Changes in Policy and Resources
Figure 1: "Measure-Act-Measure" Mechanism for Evaluating
Solid Waste Activities
-------
types of indicators — effectiveness, efficiency, and output mea-
sures—are not unrelated. For this reason, changes in policy
variables or in the level of resources that appear warranted on
the basis of effectiveness measures should always be reviewed
in light of their potential effect on system performance and system
productivity.
A DESCRIPTION OF HOW WE PERFORMED THE PROJECT
The project was performed in several phases that consisted
of the activities summarized below:
Phase I — Collection and Review of Background
Materials. A review of the literature on effective-
ness measurement systems was undertaken and the
solid waste managers in each of the 10 largest cities
were contacted to determine the types of effective-
ness measures, if any, that their communities were
using. Interviews and discussions were also held with
numerous persons involved in solid waste management
activities to obtain their views on the relative impor-
tance and usefulness of the various measures.
Phase II — Methodology Development. An analytical
framework was developed from which a comprehen-
sive set of effectiveness indicators was derived, along
with methods for collecting and recording the data.
Additionally, the various combining methods were re-
viewed and a procedure for developing composite mea-
sures was selected.
Phase ni — Field Testing of the Measurement Meth-
odology. A field demonstration, designed to assess
the measurement methodology and its implementation
in an urban community was conducted. The City of
Baltimore was used as the field test site.
Phase IV—Analysis of the Field Test Data. The field
test data were analyzed to assess the feasibility of
producing the measures, the consistency or reliability
of the subjective-type measurements, the correlations
among the measures, and the feasibility of producing
composite measures of effectiveness for solid waste
activities.
-------
ORGANIZATION OF THE REPORT
The report is organized into seven chapters and covers the
activities associated with the phases listed above. The first chap-
ter provides introductory material relevant to the project. The
second chapter describes the findings that emerged from a review
of background and other materials on existing methodologies de-
signed to measure the effectiveness of solid waste operations.
The development of the methodological approach and combining
techniques used in the project are described in the third and fourth
chapters. Details on the field demonstration comprise the fifth
chapter. The last two chapters summarize the findings and rec-
ommendations that were developed on the basis of the field data.
In addition, the report includes six appendices. These pro-
vide additional information on the methodological approach, survey
design, and field test data collection forms and procedures that
were used in the project. They include a description of how the
analysis of the field test data was performed, as well as detailed
tables, charts, and graphs to support some of the findings pre-
sented in the body of the report.
-------
II. BACKGROUND MATERIALS REVIEW
A number of measurement systems have been developed in
recent years to assess the effectiveness of solid waste operations.
Some of these systems or variations thereof have been implemented
in several of the larger cities across the nation. This chapter de-
scribes the general types of measurement methodologies that have
been proposed to collect effectiveness-type information, their re-
spective uses in the 10 largest cities in the country, and the rela-
tionship of this project to existing measurement systems.
TYPES OF MEASUREMENT METHODOLOGIES
The measurement methodologies developed during the past
few years generally include one or more of the following elements:
Direct assessment of conditions, based upon physical
measurement or direct observation by trained persons.
Indirect assessment of conditions, based upon data
obtained from records or ledgers.
Citizen perception of conditions, as indicated in com-
plaint records or special attitudinal surveys.
These measurement methodologies are discussed in the subsections
that follow.
Direct Assessment of Conditions
Nearly all of the proposed measurement systems call for
direct measurement of existing conditions, particularly along
streets and alleys in the community. However, for the most
part, only the overall aesthetic conditions of these areas are con-
sidered when measurements are made. Although provision is
generally made within most systems for collecting information on
other items of interest (e.g., bulky wastes), there is, as a rule,
no attempt to include these other items as formalized measures
per se_ (e. g., the number of discarded bulk items per square mile).
-------
The existing methods for directly assessing conditions fall
into one of two categories:
• Measurement of the area covered by litter
• Measurement of the volume of litter in a given area.
These methods are described in the following paragraphs.
Measurement of the Area Covered by Litter
Two techniques have been proposed for measuring the
area that is covered by litter. One scheme, called a Visual
Inspection System, was developed by the Urban Institute.1
The other scheme, called a Photometric System, was de-
veloped by the American Public Works Association, in con-
junction with Keep America Beautiful.2
The Visual Inspection System consists of a set of pro-
cedures whereby a trained inspector gives a numerical
rating on a scale of 1 to 4 to the litter conditions observed
on a street or alley. A set of photographs, scaled to illus-
trate the range of litter conditions, is used as a standard
in making these ratings. Whenever the conditions on streets
and alleys fall in between the conditions illustrated in the
photographs, intermediate rating points of 1.5, 2.5, and 3.5
are utilized. The originators of this scheme recommend
that inspectors, along with rating the cleanliness and appear-
ance of streets and alleys, note factors such as the presence
of abandoned automobiles and health and fire hazards, so
that corrective action may be taken.
The Photometric System is an attempt to develop a
more objective way of measuring aesthetic conditions in
streets and alleys. Under this method, photographs are
taken of the actual solid waste accumulations at random
points along a sample of streets and alleys. The camera
must be placed so that all photographs are taken at the
same angle, from the same distance, and represent a 6- by
16-foot rectangular area.
-------
After the pictures are developed, a clear plastic over-
lay, marked off into 96 grids, is placed over each picture.
The grid overlay is designed so as to have the same per-
spective as the photographs. This is necessary in order to
compensate for the fact that things in the foreground tend to
look larger and more important than those in the background.
By counting the number of grids that contain litter^ the total
area covered by litter can be determined. The originators
of this scheme recommend that the number of grids contain-
ing litter be converted to the following six-point rating scale:
Number of Grids
Rating Containing Litter
1 0-4
2 5-10
3 11-20
4 21-30
5 31-40
6 41 or more
Both of these two procedures for assessing the area
covered by litter have certain inherent problems. The
visual inspection procedure relies heavily on a subjective
assessment of conditions, and, as such, is subject to in-
spector biases and inconsistencies. The photometric pro-
cedure, while it has much to recommend it, requires that
a substantially greater amount of time be spent for its im-
plementation. Because the pictures must all be taken from
the same perspective, a fair amount of set-up time is re-
quired. Second, the area photographed must be one that is
free of automobiles for at least 25 to 30 feet. This often
poses difficulties when it is desired to obtain a second mea-
surement to assess changes in conditions. Third, difficulties
may be encountered in the actual counting of the littered
grids because: dark pieces of trash are not easily distin-
guishable in shadows; wet areas cause a glare to the picture,
making some of the trash indistinguishable; areas around
broken pavement are often discolored and appear as though
they might be litter; and, there is human error in counting
the number of littered squares. Fourth, the photographs are
not readily interpretable without a magnifying glass.
-------
Measurement of the Volume of Litter
As an alternative to measuring the area covered by
litter, a technique has been developed for measuring the
volume of litter in a given area. This technique, developed
by Ralph Stone and Company, Inc., requires that all of the trash
along a street or alley be swept up, carried away, and then
measured.3 This scheme, while it removes much of the
subjectivity associated with measuring the overall appearance
of an area, is a fairly costly scheme to implement on an
ongoing basis, as well as one that requires the absence of
parked cars, trucks, and other vehicles along the street.
Indirect Assessment of Conditions
A few of the proposed measurement schemes suggest utilizing
information on area conditions available from records and ledgers
to assess the effectiveness of solid waste activities. Some of the
proposed indicators include the number of trash fires from solid
waste accumulations, the level of external rat infestation, the
number of missed collections, and so forth.
Formulation of these measures generally requires a fair
degree of cooperation among the personnel in different municipal
government agencies (e. g. , health department, fire department,
sanitation department, housing department, etc. ). This is be-
cause the data that are needed are not available solely within the
sanitation agency. Moreover, the data, when accessible, are not
always comparable. For example, the sanitation department may
collect and summarize information using sanitation districts as a
basis, while the health and fire departments may utilize different
service units (more suited to their own internal needs) in their
summarization of the data. Additionally, even when the data bases
are comparable, the level of summarization may be too aggrega-
tive to allow comparisons except among large geographic areas
within a city.
For these reasons, implementation of a measurement system
that utilizes available data will generally require a revision of an
existing management information system or the development of a
new information system. It will also require a system which en-
compasses involvement by more than just the sanitation department,
because of the variety of data that would be included.
10
-------
Citizen Perception of Conditions
Some of the proposed measurement schemes suggest using
citizen input to assess the effectiveness of sanitation activities.
Citizen input can be obtained from:
• A systematic review of complaint records on file with
the sanitation agency.
• A special survey of citizens to obtain their attitudes
about sanitation activities.
Complaint data are useful in pinpointing short-term or ex-
treme problems. Long-standing deficiencies are generally less
detectable from complaint data for two reasons: (1) persons
having complaints may "give up" if remedial action is not initiated
early on; and, (2) persons may become so accustomed to the
deficiencies that they do not register complaints.
One of the problems associated with using complaint records
is that generally only a handful of citizens bother to file formal
complaints. Because the views of these persons may not be rep-
resentative of the community-at-large, there are likely to be
biases associated with using complaint data to measure the overall
effectiveness of sanitation activities. A carefully designed citizen
attitude survey will, as a rule, provide more accurate and reliable
information about citizen perception of conditions than will volun-
teered complaints.
Citizen attitude surveys can be conducted by interviewing a
sample of residents either in-person, over the telephone, or
through the mail. The choice among survey methods depends on
the desired degree of accuracy and the amount of money the com-
munity is willing to spend. The proponents of citizen attitude
surveys recommend that data be gathered concurrently on citizen
satisfaction with other municipal services as well as solid waste
operations.
11
-------
EFFECTIVENESS MEASUREMENT SYSTEMS IN THE TEN
LARGEST CITIES
Summary information on the effectiveness measurement
systems in use in the ten largest cities in the nation was collected
at an early point in the study through telephone and personal inter-
views with sanitation department personnel. Where available,
documentation related to the measurement systems was also ob-
tained.
The review of measurement systems in use revealed the
following:
Only two of the cities — New York and Washington,
D. C. —have implemented formalized procedures
for systematically assessing the effectiveness of
their solid waste operations. A third city, Balti-
more, is planning to implement such a system in
the near future. In all three cases, the assessment
is based on the Visual Inspection System. All three
measurement systems are funded by sources outside
the sanitation department.
In the remainder of the seven cities that were con-
tacted— Cleveland, Chicago, Dallas, Detroit, Houston,
Los Angeles, and Philadelphia—there are no formal-
ized procedures for measuring the effectiveness of
sanitation operations. Crew foremen and supervisors
periodically inspect areas subsequent to collection and
make a qualitative assessment of conditions, but no
quantitative measures are utilized.
The effectiveness measurement systems for New York City,
Washington, D. C. , and Baltimore, are described below.4'5'6
New York City
Visual inspection procedures for rating the overall cleanli-
ness of designated areas are being used in the City of New York.
This system, called Project Scorecard, utilizes the techniques
developed by the Urban Institute, with some modifications. Score-
card utilizes an eight-point scale, where 1.0 is immaculately clean
12
-------
and 4.5 is filthy. The scale progresses at .5 intervals to reflect
intermediate levels of cleanliness.
Under the Scorecard system, an inspector rates both sides
of the street and the adjoining sidewalks, but not the alleys. The
location of abandoned cars and bulk trash items are noted. These
are reported to the sanitation district superintendent responsible
for the area so that remedial action can be taken.
The streets and sidewalks are assigned separate ratings,
which are then averaged together to form a single rating for each
block inspected. The block averages are subsequently combined
and a new composite average is produced for what is called a
"strip" (a linear set of blocks). These strips form the basic unit
on which comparisons are made between sanitation districts.
The ranking of sanitation districts, based on the cleanliness
ratings, has resulted in minor competition among the various
district superintendents to keep their areas as clean as possible,
as none of them want to appear at the bottom of the list.
At the time that the information on Project Scorecard was
obtained, the system was operational on Manhattan's Lower East
Side and in areas designated by the City as needing concentrated
cleaning attention. Inspections were being made several times
weekly along all streets where the system was operational. Plans,
however, were underway for expanding Scorecard's operations to
permit inspection of a sample of streets in the entire city.
Washington, D. C.
Washington, D. C. , has reinstituted a Visual Inspection
System for assessing the appearance of streets and alleys within
the city. A nine-point scale is used, with 1.0 representing the
cleanest areas and 5.0 representing the dirtiest areas. The scale
progresses at half-point intervals to reflect conditions in between
the two extreme values.
The inspectors rate one side of the street, both sides of
alleys, conditions on public property, and conditions on private
property. Information is also recorded about dead animals,
abandoned vehicles, clogged catch basins, bulky waste, fire
hazards, and a number of other items of interest.
13
-------
The entire city is inspected four times a year^ and the in-
formation is summarized by census tract, by collection route,
and by service area. In "clean" areas of the city, 20 to 30 per-
cent of the blockfaces (the area along one side of the block from
the center line of the street to the curb) and adjacent public and
private ways are inspected. These are selected on the basis of
random sampling techniques. In the "not so clean" areas, 80 to
100 percent of the blockfaces and the adjacent public and private
ways are inspected. All alleys in the city are inspected, inde-
pendent of whether the area is classified as clean or not.
The unique aspect of the Washington, D. C., system is that
it is one of the few systems where the information that is collected
is assimilated into an overall rating to produce an "Environmental
Rating" for an area. A 100-point rating scale is used. Conditions
on both public property and private property are included in formu-
lating the index. The various index components and relative weights
are as follows:
Weight (in terms of
Item or Condition points assigned)
Overall Blockface or Alley
Cleanliness Rating 40
Blockface or Alley Cleanliness
Rating with Leaves 10
Unsightly Conditions on the
Public Way 10
Unsightly Conditions Beyond
the Public Way 20
Presence of Special Items
(e.g., dead animals, abandoned
vehicles, bulk, etc.) 20
TOTAL 100
Based on the weighting scheme shown above, a composite
rating is developed for each blockface or alley that is inspected.
The individual composite ratings are averaged to give an overall
rating for the area under consideration.
14
-------
Baltimore
Baltimore is currently in the process of implementing a
solid waste effectiveness measurement system. The system
combines- effectiveness measures with information on solid waste
activities and costs, in an attempt to identify the types and costs
of programs and resource combinations that have the greatest
effectiveness. A visual inspection process on cleanliness and
appearance of streets and alleys, along with a procedure to obtain
information on sanitation violations and citizen complaints, con-
stitute the effectiveness measurement system in Baltimore. The
effectiveness measures will form part of a comprehensive sani-
tation management information system.
Baltimore has experimented with the Photometric System
to produce cleanliness ratings and has decided that the visual in-
spection procedure is the more efficient method to use for their
purposes for the following reasons. First, they experienced
difficulties in finding the designated areas free from parked
vehicles at the time the photographs were taken and in interpre-
ting the photographs, even with a magnifying glass. Second, they
found the method could not be applied during periods of inclement
weather. Thirdly, they felt that the increased degree of accuracy
(which the Photometric System provides) was not sufficient enough
to justify the high costs associated with operating this type of
system.
GENERAL DEFICIENCIES IN EFFECTIVENESS MEASUREMENT
SYSTEMS
The measurement methodologies presented in this chapter
have proven useful in solid waste decision-making, as evidenced
by their implementation and continued use in several cities. How-
ever, there are certain weaknesses in these measurement systems
that necessitated the comprehensive investigation undertaken during
this project. These weaknesses may be summarized as follows:
The major thrust of these systems is on solid waste
cleaning and collection activities as illustrated by
the emphasis upon measures to rate cleanliness and
appearance of streets and alleys. The storage func-
tion is considered only to a limited extent in that
indicators on the condition of cans are included in the
Baltimore and Washington measurement systems. No
attention at all is given to the transportation function.
15
-------
The primary focus of these systems is on measuring
the appearance of the area. Although provision is
generally made for collecting ancillary information,
routine measurement of factors such as health and
safety hazards is generally not part of the measure-
ment system. The potential impact of solid waste
systems upon these factors may be considerable;
their effects may, in fact, be opposite to their effects
upon the more easily measured variables such as street
cleanliness and appearance. Therefore, a compre-
hensive measurement system must include indicators
for these factors.
The systems generally do not provide a means for com-
bining the individual measures into an overall effec-
tiveness rating. The schemes being implemented in
New York and Baltimore, provide indices of overall
street and alley cleanliness, but that is the extent to
which individual measures are combined. In Washing-
ton, D. C., where a composite rating of effectiveness
has recently been developed, cleanliness factors re-
ceive 80 percent of the total weight. Without usable
methods for combining the various solid waste indicators
into overall measures of effectiveness, it is difficult
to judge the true status of a community's solid waste
operations or to establish comparisons with other
communities.
The systems are essentially "parochial" in that they
were designed for specific communities to serve
specific purposes. Although guidelines are generally
provided for the extention of these systems to other
cities, the selection of the various measures and the
measurement techniques is influenced by the persons
who originally developed the measurement methodologies.
This project, while building upon the available measurement
systems, attempted to overcome the deficiencies cited above.
Measures related not only to collection and cleaning activities but
also to storage and transportation activities were considered. A
fairly comprehensive set of indicators, encompassing health hazards,
safety hazards, fire hazards, etc., was developed and applied during
a field test. A method for combining disparate indicators into single
16
-------
effectiveness measures was developed and implemented using the
data collected during the field survey. Finally, a set of procedures
was designed to assist communities that want to develop their own
unique measurement schemes, rather than adopt existing ones.
17
-------
III. DEVELOPMENT OF MEASURES
AND MEASUREMENT TECHNIQUES
There were two major aspects to the methodology develop-
ment phase of the project — (1) the development of individual mea-
sures and measurement techniques for assessing the effectiveness
of the solid waste operations of storage, collection, and local
transportation, and (2) the development of analytical procedures
for combining individual measures to produce an overall measure
of effectiveness. This chapter describes how the measures and
the measurement techniques were developed and provides a com-
plete list of the "candidate" effectiveness measures, many of
which were investigated during the pilot test phase of the project.
The following chapter describes the development of the combining
methods.
The procedures utilized in developing a comprehensive set
of effectiveness measures may be summarized as follows:
• Definitions were developed for the solid waste operations
included in the project.
• The scope of the study was defined relative to the various
generic types of effects that could be measured.
• An analytical framework, consisting of measurement
categories and indicators matched to the solid waste
operations under study, was developed.
The remaining paragraphs discuss these procedures in greater detail.
TERMS RELEVANT TO THE PROJECT WERE DEFINED
Solid waste materials are thos-e unused or unwanted materials
that result from normal community activities and have insufficient
content to be free flowing. The development of policies and proce-
dures for controlling the generation, storage, collection, transport,
19
-------
separation, processing, recovery, and disposal of these materials
is the responsibility of those federal, state, and local officials who
oversee and operate the solid waste management system. This
project focussed specifically on the effectiveness of policies and
procedures related to the storage, collection, and local transport
of solid waste materials.
The components of the solid waste management system that
this project focussed on were defined as follows:
Storage Operations — The methods by which solid waste
products are discarded by individuals, households, and
establishments, in either a loose or a contained form,
prior to collection/removal by public or private haulers.
Collection Operations — The methods associated with
the gathering or accumulating of solid waste products
from various storage points for the purpose of trans-
porting them to a central waste depository site; i. e.,
a transfer station, reclamation center, or disposal site.
Local Transportation Operations — The methods asso-
ciated with the conveyance of solid waste products
from a final collection point to the closest transfer
station, reclamation center, or disposal site.
Storage practices thus reflect the mode in which solid waste
materials initially present themselves to the system. This maybe
in a containerized form (from households, commercial establish-
ments, etc. ) or in a non-containerized form (such as street litter,
bulk waste, etc. ). Collection activities include: regular mixed
refuse collection, bulk or special item pickup, and street and alley
cleaning. Transportation practices refer solely to activities asso-
ciated with the conveyance of solid waste products to the closest
depository site. Long haul transportation via rail, truck, and so
forth, was not part of this project.
These solid waste activities were considered relative to
municipal solid waste products only; i. e., materials discarded
by households, apartment complexes, and commercial concerns,
and materials found in public areas such as streets, sidewalks,
20
-------
alleys, vacant lots, parks, etc. The types of solid waste materi-
als likely to be generated by these sources include:
• garbage (not discharged into the sewer system)
• rubbish (both combustible products, such as paper,
leaves, wood; and non-combustible products, such
as metal, glass, plastic, and so forth)
• ashes
• street refuse (street sweepings, dirt, leaves, catch
basin dirt, street litter, and so forth)
• bulky wastes
• abandoned vehicles
• construction and demolition waste.
ONLY THE EFFECTS OF SOLID WASTE ACTIVITIES ON THE
PUBLIC WERE CONSIDERED
Conceptually speaking, there are two distinct types of effects
which may be expected to result from the operation of a solid waste
system, namely:
Effects which are external to system operating pro-
cedures
Effects which are internal to system operating pro-
cedures.
Effects external to the system refer to the benefits and
damages which are sustained by the public as a result of solid
waste system operations. An example of this type of effect is
unsightliness caused by the presence of spilled or scattered
refuse subsequent to mixed refuse collection.
21
-------
Effects internal to the system refer to the benefits and
damages that are sustained by the system itself as a result of a
given type of solid waste operation or procedure. An example of
this type of effect would be injuries suffered by waste collectors.
This project concerned itself only with the former type
effect; i. e., effects external to the system.
A GENERAL FRAMEWORK WAS DEVELOPED
Once the focus of the study had been clearly defined, a
general framework was developed from which a "candidate" set
of effectiveness measures could be derived. The framework con-
sisted of:
• Major measurement categories that corresponded to
the objectives of a solid waste system.
• Qualitative indicators (or descriptors), matched to
the solid waste operations under study, for each mea-
surement category.
• Quantitative measures (or variables) for each indi-
cator.
• Measurement techniques for each measure.
An overview of the general framework is presented in Figure 2 on
the following page. The elements of the framework are described
in the paragraphs that follow.
Several Major Categories for Measuring Effectiveness Were
Developed to Reflect the Objectives of a Solid Waste Management
System
In general, effectiveness measures relate to the degree to
which goals and/or objectives are being met. Thus, in order to
properly assess the effectiveness of the solid waste system, it is
necessary to define its goals and/or objectives in a precise manner.
22
-------
Category A
Category B
1 1 I
Hazards Hazards Hazards Hazards
Due to Due
Poor Poor
Sto
to Due to Due to
Poor Poor
rage Collection Transport Storage
|
Major
Measurement
Categories
Indicators by
Category Matched
to Solid Waste
Operations
Measure Measure Measure
Number Number Number
1 2 3
Method Method
of of
Measur- Measur-
ing ing
Measure
Number
1
Measures
for the
Indicators
Measurement
Techniques
Figure 2: Overview of the General Framework for
Developing Candidate Effectiveness Measures
-------
Since the project was not meant to deal with a specific type
of solid waste system, but rather with solid waste systems in
general, the statement of objectives had to be broad enough to
apply to any solid waste system, independent of its physical com-
ponents. The statement of objectives that was developed may be
summarized as follows:
The objective of a solid waste management system is
to provide for the operation of a waste handling system
with a level of service sufficient to —
(1) meet the community needs in terms of handling
and disposing of all unused or unwanted solid
waste materials that are discarded within the
community
(2) maintain a safe, healthful, and aesthetically
pleasing community; i. e., counter those effects
which would result from an unmanaged or im-
properly functioning solid waste system.
Based on this statement of objectives, it can be inferred
that an effective solid waste system is one which:
• meets the community's storage and collection needs
• mitigates against conditions that might cause deterio-
ration of public health
• acts to prevent accumulations of solid waste materials
which might cause injury to the public
• strives to minimize deterioration of the appearance of
the community due to street litter and solid waste
materials
• acts to prevent offensive odors caused by inadequate
storage or infrequent collection of solid waste materials.
24
-------
Thus, the following effectiveness categories emerge from
the statement of objectives:
• Meeting the Community's Storage and Collection Needs
• Safety of the Public
• Public Health
A j.i x- /-. T.L- J — Appearance
• Aesthetic Conditions < _^
I — Odor
These effectiveness categories relate directly to the reasons for
performing solid waste collection and transportation activities;
for example, keeping the streets clean and preventing public
health and safety problems from occurring. They pertain to solid
waste systems in general.
In order to accomplish the objectives stated above, each
solid waste system sets its own ordinances, standards, and
policies with respect to its storage, collection, and transporta-
tion operations. These standards and procedures generally relate
to the types of allowable containers and to the frequency of collec-
tion and cleaning operations. Thus, the degree of compliance with
these standards constitutes another major category against which
effectiveness can be measured. That is, if plastic bags as outside
storage containers are prohibited, then measurement of the com-
pliance level for this could be important with respect to the overall
effectiveness of the solid waste system.
In addition, there are features of the actual solid waste
system in use (i. e., its equipment, labor use, and other methods
and procedures) that are either closely related to meeting the
overall objectives or are constraints (restrictions) on how the
job of storing, removing, and transporting refuse should be
accomplished. Some of these features may cause inconvenience
or discomfort to the public. Collection truck noise levels and
induced traffic congestion because of collection scheduling are
examples of these types of operating system features.
25
-------
Thus, for a specific solid waste system, two additional
categories for assessing effectiveness may be set forth:
Compliance with System Standards
Inconvenience/Discomfort to the Public Because of
System Operating Procedures.
Table 1 shown below summarizes the major measurement
categories that were developed for this project.
Table 1. SUMMARY OF MAJOR CATEGORIES
FOR MEASURING EFFECTIVENESS
CATEGORIES OF EFFECTIVENESS RELATED TO NATURE
OF SOLID WASTE SYSTEMS IN GENERAL:
• Public Health
• Public Safety
• Appearance of Community
• Odor Within Community
• Satisfaction of Community's Storage and
Collection Needs
CATEGORIES OF EFFECTIVENESS RELATED TO SOLID
WASTE SYSTEM UTILIZED:
• Compliance With Standards
• Inconvenience/Discomfort to Public
26
-------
Indicators of Effectiveness Matched to Solid Waste Operations
Were Developed for Each Category of Effectiveness
For each of the categories of effectiveness listed in Table 1,
a number of indicators were developed. These indicators were
designed to provide a qualitative statement of conditions from
which quantitative measures (or variables) could subsequently be
developed. That is, they were used to describe the conditions
that we would be attempting to measure.
In developing these indicators, an attempt was made to match
effectiveness categories with the specific solid waste operations
under study; namely, storage, collection, and transportation.
This was done in order to facilitate the formulation of quantitative
measures that related specifically to one or more of the operations.
A summary of the indicators of effectiveness, matched to
solid waste operations, is shown in Table 2 on the following page.
Quantitative Measures and Measurement Techniques Were Devel-
oped for Each of the Indicators of Effectiveness
For each effectiveness indicator, measures (or variables)
were developed for which data would be gathered. This initial
list of variables was subsequently reviewed by an in-house team
of analysts, and a number of measures were eliminated from
further consideration because they did not meet one or more of
the evaluative criteria set forth.
The evaluative criteria used to screen the initial list of
variables included an assessment of each measure's:
Validity—Does the measure indicate what it purports
to?
Accuracy—Will the measurement reflect a "true"
picture of the measured conditions?
Reliability (or) Consistency — Will independent ob-
servers come up with similar measurements of the
same phenomena?
27
-------
Table 2. INDICATORS OF EFFECTIVENESS FOR EACH
MEASUREMENT CATEGORY, MATCHED TO
SOLID WASTE SYSTEM OPERATIONS
CO
0)
c
0
c
OJ
in
>
-------
Usefulness to Solid Waste Managers — Will the mea-
sure assist decision-makers in making comparisons
among areas and/or in analyzing trends over time?
Measurement techniques were then designed for each of the
resultant measures. These related to: (1) the type of data that
would be needed to formulate a given measure; (2) the procedures
and methods that could be used to obtain the requisite data; and
(3) the suggested frequency with which the data should be collected
and the measures formulated.
The full set of candidate effectiveness measures and their
respective measurement techniques is presented in tabular form
in Table 3 on pages 30 through 36. This list is designed to indi-
cate the relevant measurement category, solid waste activity,
and effectiveness indicator associated with each measure. The
table is organized as follows:
• Measurement Category — Indicates the general cate-
gory of effectiveness to which the measure relates.
• Solid Waste Activity—Delineates the solid waste
activity (storage, regular collection, special collec-
tion, cleaning, transportation) to which the measure
relates.
• Indicators of Effectiveness — Designates the effective-
ness indicator to which the measure relates.
• Measures of Effectiveness—Indicates the form of the
measure itself.
• Measurement Techniques — Specifies what would be
measured, how the data would be collected, and how
frequently measurements should be made.
• Type of Measure—Indicates the source of data by
generic category: direct observation of existing con-
ditions, special measurement apparatus, available
records and ledgers, or a household survey.
29
-------
Table 3. CANDIDATE EFFECTIVENESS MEASURES AND
MEASUREMENT TECHNIQUES BY MEASUREMENT
CATEGORY, INDICATOR, AND ACTIVITY
MEASUREMENT CATEGORY - PUBLIC HEALTH
Solid Waste
Activity
Storage
Regular
Collection
Cleaning
Indicators of
Effectiveness
Conditions
hazardous to
health in storage
areas/or private
areas
Widespread health
hazards through-
out neighborhood
storage areas
Health hazards in
waste collection
area
Health hazards on
the public way
(streets and alleys)
Health hazards on
vacant lots and
public areas
Measures of Effectiveness for
Each Indicator
—Average rat count for storage areas
—Percent of storage areas where rat
count exceeds given threshold
A f 1
—Percent of storage areas where fly
count exceeds given threshold
—Percent of storage areas found to con-
tain health hazards
—Average health hazard rating for stor-
age areas
—Percent of storage areas where health
hazard rating exceeds a given threshold
—Percent of blocks where more than "X"
percent of the storage areas are found
to contain health hazards
—Number of containers per block set out
the night before collection which pose
potential health hazards
—Percent of inspections found to contain
health hazards on the public way
(inspection unit = blockface or alley)
-Percent of inspections where health
hazards are found on lots or public
areas (inspection unit = blockface or
alley)
MEASUREMENT TECHNIQUES
Measurement Elements
Rat count for each stroage area inspected
Fly count for each storage area inspected
Presence of health hazards in waste storage
area 6 = yes, 7 = no
Types of Health Hazards
— Flies, insects
—Signs of rodents
—Garbage not in metal container
—Garbage in a metal container without
tight lid
— Decaying animals
—Other health hazard
Health hazard rating assigned to each storage
area inspected. 3 point scale (based on above
data)-
1 - no health hazards observed
2 = minor health hazards observed (no
signs of insects, flies, or rodents)
3 = major health hazards observed (signs
of insects, rodents, and flies)
This is a derived measure based on data
gathered from inspection of several waste
storage areas in a number of blocks
Number of containers in waste collection
areas which are of a nature that rats and/or
animals can gain access and which are put
out the night before collection
Presence of health hazards (as defined above}
on streets and alleys. 6 = yes; 7 = no
Presence of health hazards (as defined above)
on lots and public areas' 6 • yes; 7 = no
Data Collection Methods
Inspection of storage areas using rat
count apparatus
i fl
count apparatus
Inspection of storage areas using a
checklist to denote specific health
hazards observed
Inspection of waste collection areas
Inspection of streets and alleys using
checklist to denote specific health
hazards observed
Inspection of lots and public places
using checklist to denote specific health
hazards observed
Frequency of
Collection3
Q
Q
Q
Q
Q
Q
Type of
Measure b
SM
SM
DO
DO
DO
DO
See footnotes at end of table.
-------
Table 3 (continued)
MEASUREMENT CATEGORY - PUBLIC SAFETY
Solid Waste
Activity
Storage
Regular
Collection
Cleaning
Indicators of
Effectiveness
Conditions
hazardous to
safety in storage
areas/or private
areas
Widespread
safety hazards
throughout
neighborhood
storage areas
Safety hazards
on the public
way {streets and
alleys)
Safety hazards
on vacant lots
and public
areas
Measures of Effectiveness for
Each Indicator
—No. of fires caused partially by improper
storage of combustible solid waste per
1000 persons
—Percent of storage areas found to con-
tain safety hazards
—Average safety hazard rating for stor-
age areas
—Percent of storage areas where safety
hazard rating exceeds a given threshold
—Percent of blocks where more than "X"
percent of the storage areas are found
to contain safety hazards
—Percent of inspections found to contain
safety hazards on the public way
(inspection unit = blackface or alley)
—Percent of inspections where safety
hazards are found on lots or public
areas (inspection unit = blockface or
alley)
MEASUREMENT TECHNIQUES
Measurement Elements
Number of fires attributable to solid waste
accumulation
Presence of safety hazards in waste storage
area: 6 = yes, 7 = no
Types of Safety Hazard
—Broken glass
-Barbed wire
— Refrigerator with door mtact
—Combustible waste sufficient to cause
a fire
—Other safety hazards
Safety hazard rating assigned to each storage
area inspected: 3 point scale (based on above
data) --
1 = no safety hazards observed
2 = minor safety hazards observed (i e.,
broken glass, barbed wire)
3 = major safety hazards observed (i.e.,
refrigerator with" door intact, waste
sufficient to cause fire)
This is a derived measure, based on data
gathered from inspection of several'waste
storage areas in a number of blocks
Presence of safety hazards (as defined above)
on streets and alleys' 6 = yes; 7 = no
Presence of safety hazards (as defined above)
on lots and public areas: 6 = yes; 7 = no
Data Collection Methods
Review of fire department records
Inspection of storage areas using a check-
list to denote specific safety hazards
observed
Inspection of streets and alleys using a
checklist to denote the specific safety
hazards observed
Inspection of lots and public places using
a checklist to denote the specific safety
hazards observed
Frequency of
Collection a
A
Q
Q
Q
Type of
Measure D
R
DO
DO
DO
See footnotes at end of table.
-------
Table 3 (continued)
MEASUREMENT CATEGORY - APPEARANCE OF COMMUNITY
Solid Waste
Activity
Storage
Regular
Collection
Special
Collection
Cleaning
Transportation
Indicators of
Effectiveness
Unsightliness of
storage areas/or
private areas
Presence of bulk
items in storage
areas/or private
areas
Spilled or scatter-
ed refuse
subsequent to
collection
Presence of
abandoned bulk
items
Presence of
abandoned
automobiles
Unsightliness of
public way
subsequent to
cleaning
Unsightliness of
vacant lots and
public areas
Clogged drain
basins
Spilled or
scattered refuse
during transport
Measures of Effectiveness for
Each Indicator
—Average appearance rating for storage
areas
—Percent of storage areas where appear-
ance rating exceeds a given threshold
—Percent of storage areas found to con-
tain abandoned or discarded bulk items
—Percent of blockfaces containing
spilled or scattered refuse subsequent to
collection (where curbside collection
is performed)
+
Percent of alleys containing spilled or
scattered refuse subsequent to collec-
tion (where alley collection is perform-
ed}
—Percent of inspections where abandoned
or discarded bulk items are observed
(inspection unit = blockface or alley)
—Percent of inspections where abandoned
automobiles or trucks are observed
(inspection unit = blockface or alley)
—Average litter for streets and alleys
—Percent of streets and alleys where litter
rating exceeds a given threshold
—Number of unsolicited complaints from
citizens about the appearance of their
community per 1000 persons
— Index of citizen satisfaction with the
appearance of their community
—Average litter rating for vacant lots and
public areas
—Number of drain basins which are clog-
ged per basin inspected
—Percent of collection fleet likely to
cause spillage while in transport
MEASUREMENT TECHNIQUES
Measurement Elements
Appearance rating assigned to each storage
area inspected: 7 point scale
Presence of abandoned or discarded bulk
items in storage area: 6 = yes; 7 = no
Presence of spilled or scattered refuse
subsequent to collection: 6 = yes; 7 = no
Presence of abandoned or discarded bulk
items on streets, alleys, lots, and public
areas: 6 = yes; 7 = no
Presence of abandoned automobiles or
trucks on streets, alleys, lots, and public
areas: 6 = yes; 7 = no
Appearance rating assigned to each street or
alley inspected: 7 point scale
Number of complaints from residents about
the appearance of their community
Citizen attitudes toward general appearance
of their commupity: 4 point scale
Appearance rating assigned to each vacant
lot or public place inspected' 7 point scale
Presence of clogged drain basin: 6 = yes;
7 - no
Number of open trucks in the collection
fleet
Data Collection Methods
Visual inspection of storage areas, using
photographs as a reference for selecting
the appropriate rating
Inspection of storage areas
Inspection of waste collection areas
Inspection of streets, alleys, lots, and
public places
Inspection of streets, alleys, lots and
public places
Visual inspection of streets and alleys,
using photographs as a reference for
selecting the appropriate rating
Review of sanitation department records
Survey of a representative sample of
residents to ask about appearance of the
community
Visual inspection of lots and public places
using photographs as a reference for
selecting the appropriate rating
Inspection of streets and alleys
Review of records on truck inventory
Frequency of
Collection a
Q
Q
Q
Q
Q
Q
M
A
Q
Q
A
Type of
Measure0
DO
DO
DO
DO
DO
DO
R
S
DO
DO
R
See footnotes at end of table.
-------
Table 3 (continued)
MEASUREMENT CATEGORY - ODOR
Solid Waste
Activity
Storage
Regular
Collection
Indicators of
Effectiveness
Offensive odors
in waste storage
areas
Widespread odors
throughout
storage areas
Measures of Effectiveness for
Each Indicator
—Percent of storage areas found to con-
tain offensive odors
—Percent of blocks where more than "X"
percent of the storage areas are found
to contain offensive odors
—Number of unsolicited complaints from
citizens about the existence of offen-
sive odors per 1000 persons
MEASUREMENT TECHNIQUES
Measurement Elements
Presence of odor in waste storage area:
6 = yes; 7 = no
This is a derived measure, based on data
gathered from inspection of several waste
storage areas m a number of blocks
Number of complaints from residents about
offensive odors in their community
Data Collection Methods
Inspection of storage areas
Review of sanitation department records
Frequency of
Collection3
Q
M
Type of
Measure b
DO
R
Ol
See footnotes at end of table.
-------
Table 3 (continued)
MEASUREMENT CATEGORY - SATISFACTION OF NEEDS
Solid Waste
Activity
Storage and
Regular
Storage
Regular
Collection
Indicators of
Effectiveness
"Storage —
collection capa-
"storage —
collection needs"
Capacity
deficiencies in
waste storage
areas
Widespread
capacity deficien-
cies throughout
storage areas
Measures of Effectiveness for
Each Indicator
—Amount to which the combined storage-
collection capacity falls short of the
block (in pounds of refuse}
—Percent of blocks where the combined
storage-collection capacity falls short
of the "true" storage-collection needs
—Percent of storage areas found to con-
tain an inadequate number of containers
—Percent of blocks where more than "H"
percent of the storage areas contain an
inadequate number of containers
MEASUREMENT TECHNIQUES
Measurement Elements
Amount of refuse set out for collection (and
picked up) per week as compared with the
amount of refuse generated in a given block
This is a derived measure, based on data
gathered on a number of blocks
Indications that there are an insufficient
number of containers in the waste storage
area: 6 = yes; 7 = no
—Overflowing containers
— Refuse piled on ground
—Containers in poor condition
This is a derived measure, based on data
gathered from inspection of several waste
storage areas in a number of blocks
Data Collection Methods
Review of sanitation department records;
review of census records
Inspection of storage areas
Frequency of
Collection3
Q
Q
Type of
Measure b
R
DO
See footnotes at end of table.
-------
Table 3 (continued)
MEASUREMENT CATEGORY - COMPLIANCE WITH STANDARDS
Solid Waste
Activity
Storage
Regular
Collection
Special
Collection
Indicators of
Effectiveness
Compliance with
storage
requirements
Compliance with
mixed refuse
collection
standards
Compliance with
special pickup
standards
Measures of Effectiveness for
Each Indicator
—Percent of storage areas having contain-
ers which do not comply with the
regulations
—Percent of blockfaces where mixed
refuse remains uncollected for one or
more days (where curbside collection is
performed)
4
Percent of alleys where mixed refuse
remains uncollected for one or more
days (where alley collection is
performed)
—Average delay time in meeting regular
pickup schedules for alleys/btockfaces
—Number of unsolicited complaints from
citizens about delays in pickup of mixed
refuse per 1000 persons
—Percent of instances in which there was
a delay of one day or more in the pick -
up of bulky items
—Average delay time in the pickup of
bulky items
—Number of unsolicited complaints from
citizens about delays m pickup of
special or bulk items per 1000 persons
MEASUREMENT TECHNIQUES
Measurement Elements
Presence of containers which do not comply
with the regulations: 6 - yes; 7 = no
Presence of uncollected mixed refuse along
pickup: 6 ~ no, 7 = yes
Amount of lapsed time between actual and
scheduled pickup of mixed refuse
Number of complaints from residents about
delays m the pickup of mixed refuse
Presence of uncollected bulk items on the
day subsequent to pickup: 6 = yes; 7 = no
Amount of lapsed time between actual and
scheduled pickup of bulky items
Number of complaints from residents about
delays m the pickup of bulk refuse
Data Collection Methods
Inspection of storage areas
Inspection of waste collection areas
Comparison of advertised collection
schedule with dispatchers log
Review of sanitation department records
Inspection of waste collection areas
Comparison of advertised collection
schedule with dispatcher's log
Review of sanitation department records
Frequency of
Collection3
Q
Q
M
M
Q
M
M
Type of
Measure b
DO
DO
R
R
DO
R
R
Ox]
See footnotes at end of table.
-------
Table 3 (continued)
MEASUREMENT CATEGORY - INCONVENIENCE/DISCOMFORT TO PUBLIC
Solid Waste
Activity
Regular
Collection
Transportation
Indicators of
Effectiveness
Inconvenience
due to type of
collection service
(curb, alley, etc.)
Noise from
collection
Traffic conges-
tion from collec-
tion scheduling
Property
damage from
collection
activities
Traffic congestion
during transport
to deposit site
Air pollution
from poorly
maintained
vehicles
Measures of Effectiveness for
Each Indicator
—Average amount of time spent per
household per month preparing refuse
for collection
—Percent of collection areas where noise
from collection exceeds a given thres-
hold
—Number of collection miles taking place
during early morning hours as a percent
of total collection miles
—Number of collection stops taking place
during early morning hours as a percent
of total collection stops
—Number of unsolicited complaints from
citizens about noise caused by refuse
collection activities per 1000 persons
—Number of miles of major and second-
ary arterial roads where refuse collec-
tion is performed during peak hours
— Amount of peak hour time during
which refuse collection is taking place
along major and secondary arterial
roads
—Number of reported instances of pro-
perty damage caused by collection
equipment or collection personnel per
1000 persons
—Total dollar value of property losses
caused by collection equipment or
collection personnel
—Amount of peak hour time during
which collection fleet is enroute (to
or from) central deposit source
—Percent of vehicles where air pollution
rating exceeds a given threshold
MEASUREMENT TECHNIQUES
Measurement Elements
Amount of time spent per household per
month in preparing refuse for collection
Noise measurement taken at each waste
collection area inspected
Time of day that collection takes place along
linear segments of the collection route
Time of day that collection takes pla ;e at
each collection area
Number of complaints from residents about
noise from collection activities
Time of day that collection takes place along
each major and secondary arterial road in a
given collection route
Number of reported incidents of property
damage caused by collection personnel or
collection equipment
Dollar value of individual claims made against
the sanitation department for property
damage due to collection activities
Total transport hours which occur during
peak hours of the day
Air pollution measurement taken for each
collection vehicle inspected
Data Collection Methods
Survey of a representative sample of
residents
Inspection of waste collection areas using
noise measurement apparatus
Review of dispatcher's log
Review of sanitation department records
Review of dispatcher's log
Review of sanitation department records
Review of sanitation department records
Review of dispatcher's log
Inspection of collection vehicles using air
pollution testing devises
Frequency of
Collection a
A
A
M
M
M
M
M
M
A
Type of
Measure b
S
SM
R
R
R
R
R
R
SM
CM
a*
M = monthly; Q = quarterly; A = annually.
R = records; S = household survey; DO=direct observation; SM= special measurement apparatus.
-------
IV. DEVELOPMENT OF ANALYTICAL METHODS
FOR COMBINING COMPONENT VARIABLE
MEASUREMENTS TO PRODUCE AN OVERALL
EFFECTIVENESS MEASURE
Besides developing effectiveness measures and measurement
techniques, the methodological approach to this project included:
• A review of the available methods for combining com-
ponent variable measurements into overall measures.
• The development of analytical procedures to be used
in this project to formulate solid waste effectiveness
indices.
These aspects of the methodology development are discussed in
this chapter.
RELEVANT CONSIDERATIONS IN FORMULATING EFFECTIVE-
NESS INDICES
The problem to be addressed is one of selecting a decision
model that appropriately combines a set of multidimensional vari-
ables into a unidimensional measure (index) that correctly classi-
fies the effectiveness of a community's storage, collection, and
transportation activities. The salient aspects to be considered
include:
• Types of combining models that are appropriate
• Procedures for formulating these models and deter-
mining the weights for the component variables that
comprise the index
• Methods for achieving a consensus when more than
one evaluator or decision-maker is involved.
These points are discussed in the following paragraphs.
37
-------
Available Combining Models
Fundamental to the development of an index is an underlying
model that reflects the relative importance that a decision-maker
attaches to the variables that comprise the measure. There are
several decision models that can be used to derive an index of
effectiveness. These include the linear, conjunctive, and disjunc-
tive models, as described below.
The Linear Model
The linear model produces an index that is an additive
sum of the component variables. For example, if
health, safety, and appearance are the principal indi-
cators in assessing solid waste system effectiveness,
the values of the variables associated with these indi-
cators are weighted to reflect their relative importance
and then added together to obtain an overall score.
This score is the index value for the given values of
the variables.
The Disjunctive Model
The disjunctive model produces an overall measure
that can be used to classify a solid waste system into
one of two categories — effective or ineffective. In
this model, if one of the indicators (e. g., appearance)
has an extremely high value (where high represents a
favorable condition), then the solid waste system is
deemed effective, regardless of the values of the other
variables.
The Conjunctive Model
The conjunctive model also produces an overall mea-
sure that can be used to classify a solid waste system
into one of two categories — effective or ineffective.
However^ in this model, high values on one or more
variables will not compensate for low values on the
other variables. Rather, all of the individual com-
ponent variables must meet certain minimum thresh-
olds in order for the system to be classified as
effective. For example, if the health variable did not
meet the standard, the system would be ineffective,
even if it were meeting (or surpassing) the standards
set for the other relevant variables.
38
-------
Of these three decision models, the linear model is the one
most frequently employed when a mathematical formula for com-
bining multidimensional attributes is desired. However, there is
evidence to support the contention that individuals, when faced
with the task of mentally combining potentially contrasting effects
to formulate decisions, use non-linear decision rules, similar to
those exemplified by the conjunctive and disjunctive models.8
It should be noted that in the strict sense in which these
models are defined, neither the disjunctive nor the conjunctive
model produces an index in the same manner as the linear model.
The former two models provide indices that denote "acceptability"
or "non-acceptability. " The latter model, on the other hand, pro-
duces an index that has an ordered set of values, ranging from
some minimum level to a maximum level. Thus, the linear model
can be used to rank neighborhoods. That is, if one neighborhood
has an index value of I, and another neighborhood has an index
value of I2, then I, larger than I2 implies, by the index, that the
first neighborhood is "better off" than the second neighborhood.
All that can be said when the conjunctive or disjunctive models
are used to compare neighborhoods is that certain ones are "good"
and others are "bad. " If two neighborhoods are both good or both
bad, there is no way to select the one that is better or the one that
is worse.
On the other hand, the conjunctive and disjunctive methods
explicitly require the notion of "good" or "bad" in their application,
while the linear model does not. Thus, index values derived from
the latter model do not in themselves provide sufficient information
by which to classify a neighborhood as good or bad. To assess the
status of a neighborhood using a linear model, it is necessary to
determine a threshold value for the index such that if the index
exceeds the threshold, the neighborhood is classified as good.
Otherwise, it is classified as bad.
The discussion thus far has focussed on the similarities and
dissimilarities among the three combining models, based on the
strict sense in which they have been defined. However, if mathe-
matical functions are used to approximate these various models,
This assumes that the variables are scaled so that the larger
the value of the index, the better it is.
39
-------
they all take on similar properties; that is, they are all continuous
functions that yield to the ordinal assumption, whereby rankings
can be made, and they all require explicit statements as to what
determines "goodness" or "badness" in the resultant index values.
For the linear model, there is a mathematical formulation
explicit in its definition. For the other two models, mathematical
approximations are required for their representation. Appendix A
indicates functional relationships that can be used to provide mathe-
matical approximations for the models and illustrates geometrically
the models and their approximations.
Procedures for Formulating an Index Using the Models
There are two methods that can be used to formulate indices
that are based upon the decision models described above. These
are:
Mathematical methods
Judgmental methods.
Mathematical methods are based on statistical relationships among
the measured variables. Judgmental methods, on the other hand,
utilize value judgments or opinions of one or more individuals,
knowledgeable about the activities being assessed, to formulate a
composite measure. Each of these methods is described below.
Mathematical Methods for Formulating an Index
Fundamental to the use of mathematical methods in
formulating an index is the notion that statistically signifi-
cant relationships can be established among the measured
variables. Where the actual status of that which one is
trying to classify is known or can be estimated (e. g., a
global judgment about the overall conditions in a census
tract, neighborhood, etc. ), it can be used as the dependent
variable and the profiles (as reflected in the measurements
made on health, safety, appearance) can be used as inde-
pendent variables. The relationship between the dependent
variable and the independent variables can then be estimated,
40
-------
utilizing mathematical functions that approximate the linear,
disjunctive, and conjunctive models. The model that seems
to "best fit" the data is selected. The parameters that
characterize the function become weights and reflect the
relative importance of each component variable in assess-
ing the overall effectiveness of solid waste operations.
This technique has been employed in medical studies where
the dependent variable is the presence or absence of a disease,
and the independent variables consist of symptoms, patho-
logical signs, and clinical findings prior to the time the actual
diagnosis is made.
Where there is no a priori way to characterize the
dependent variable, Principal Components Analysis may be
used as an alternative statistical procedure for combining
variables. This procedure is a method for reducing the
number of variables in such a fashion as to lose as little
information as possible.
Judgmental Procedures for Formulating an Index
As an alternative to the mathematical procedure, the
opinions of "experts" may be solicited in such a manner as to
construct a composite index of effectiveness. To form indices
corresponding to the linear, conjunctive, or disjunctive models,
the following minimal information must be obtained from the
experts:
(1) For the linear model, the judges must be ques-
tioned so as to elicit the weights w} , w2 , ...
wi for the variables x,, x2, ... xi that are to
be measured.
Generally, the weights are normalized so that
5^wj. = !• The index-of effectiveness (E) is
represented as:
E = Wj x, + w2 x2 + . . . + w- x-
41
-------
(2) For the conjunctive model, the judges must be
questioned in such a manner as to elicit the
"tolerable" thresholds t,, t2, ... tj_ for the vari-
ables x,, x2, ... x^ A neighborhood is considered
"good" if all variables exceed their respective
threshold values; i. e.,
x^tj for all i
It is considered "bad" if at least one variable
fails to exceed its threshold value; i. e., if
x^ s^ for any i
It is considered bad if none of the variables
exceed their respective thresholds; i. e., if
x
-------
Rating involves assigning weights to all of the variables
directly on a scale from zero to one, where the weights re-
flect the relative importance of the variables. For example,
the expert might assign .5 points to the health variable, .3
points to appearance variable, and so on. Ranking involves
ordering the variables in terms of their relative desirability.
This technique, however, does not permit the expert to state
the strength of his preferences.
The third method, called "forced decisions, " is based
on pairwise comparisons. When using this technique, a
decision is made for each pair of variables as to which is
the more important one. A score of "one" is assigned to the
preferred variable; a score of "zero" is assigned to the
other variable. After all pairs of variables have been
assessed, the scores for the individual variables are totalled
by variable. Weights are then determined on the basis of
the number of points a given variable receives relative to
the total points overall.
The fourth method, called Decision Alternative Ratio
Evaluation (DARE), requires the expert to assess a series
of variables in terms of how much more important one
variable is than any other. The expert is asked to take the
variables two at a time and assign a numeric value that
reflects how strongly he prefers one variable over the
other. For example, the expert might feel that the public
health variable is three times as important as the public
safety variable; he might find the public safety variable
only half as important as the appearance variable; and so
forth. This technique thus allows the expert to quantitatively
order his preferences. When the appropriate algorithim is
applied, these stated preferences can be converted into an
overall measure, where each variable receives a weight
based on its importance relative to the other variables under
consideration.
All of the above procedures belong to the general cate-
gory of decision weighting models. They are generally used
in conjunction with developing an overall measure, based on
the linear model. An alternative technique for formulating
judgmental indices is the Delphi Method, a Feedback and
Reassessment process.10 This will be discussed further in
the following subsection.
43
-------
Methods for Achieving a Consensus
Whenever more than one individual is involved in the decision-
making process, there arises what has been termed the "consensus
problem. " That is, individuals are likely to disagree on the rela-
tive importance and, in turn, the weights they assign to the vari-
ables. There is, unfortunately, no universally applicable criterion
for handling this problem, because there is no theoretical basis
for making the interpersonal comparisons that are needed in order
to appropriately combine individual preference patterns. Any
change in the method of combining the variables will affect some
persons favorably and others adversely, and there is no a priori
way of weighting the net results. This does not mean that inter-
personal comparisons should not be made. Indeed, often they
must be made. Rather, no general formula can be invented that
11 12
handles all such problems satisfactorily.
Among the proposed solutions are the following:
• Procedures or criteria for weighting the individual
opinions
• Procedures for reconciliation of individual desires
into a collective opinion.
The procedures for mechanically combining individual
opinions include both simple averages and weighted averages of
individual preferences. The simple average approach is generally
employed when there is no reason to believe that any one individual's
preferences should be given more weight than another. The weighted
average approach is applicable when dealing with expert opinion and
there is reason to believe that each individual has a specific area
of expertise. In this case, it is desirable to weight the opinions
or preferences of certain individuals more heavily than others.
The procedures for reconciliation of individual desires into
a collective opinion include Group Decision techniques and Feedback
and Reassessment techniques. Group Decision techniques involve
meeting as a group to discuss the matter, with a view to ultimately
arriving at a satisfactory compromise among the divergent views.
Feedback and Reassessment techniques involve querying each indi-
vidual separately, and then providing feedback on the opinions of
all individuals. Individuals are then asked to reconsider their
initial assessments based on the group feedback.
44
-------
The Delphi Method is the most well known form of the Feed-
back and Reassessment method. Questions are asked privately
and anonymously of each person. The distribution of responses is
summarized in a statistical fashion. The statistical summariza-
tion is then provided as feedback information to each person, who
is then asked to reconsider his position in light of the majority
response. Where an individual's opinion deviates substantially
from the group norm, he is asked to justify his reason for holding
this position. In a sense, he is asked to rate his expertise on the
question. This has the effect of causing persons without strong
preferences to move toward the views held by the majority of the
group, while allowing those with extremely strong preferences to
retain their positions on the matter. The process continues in a
similar fashion for several rounds, until fairly close accord is
reached among the participants.
THE ANALYTICAL PROCEDURES USED IN THIS PROJECT
The procedures utilized in this project to develop overall
measures of effectiveness draw upon the techniques described
above. These procedures may be summarized as follows:
(1) The field data gatherers were asked to assign a
separate composite (or global) rating to each area
they surveyed at the time they collected data on the
effectiveness variables. This composite rating was
to reflect their subjective assessment of the area and
to take into consideration only the component variables;
i. e., factors not related to the study, such as the
color of the house, were to be ignored.
(2) The relationship between the composite rating and the
component effectiveness variables was then examined,
using regression techniques. Functional forms cor-
responding to the three decision models; i. e., linear,
conjunctive, and disjunctive, were utilized to test the
nature of the relationship. The results were compared
to determine the strength of the relationship and the
relative predictability of each of the three models.
The following paragraphs further describe these procedures.
45
-------
Description of the Composite (or Global) Rating
The individuals who collected the data were asked not only
to provide information from which the individual effectiveness
measures could be formulated; but also, to provide an overall
assessment of the area. A semantic differential approach was
used in this regard. This is a self-reporting technique in which
an individual is asked to directly evaluate an attitudinal object,
and to indicate his opinion along a scale where the endpoints have
opposite meanings, such as:
most favorable : : : : : : worst
The respondent marks the scale according to how closely he feels
one adjective or the other describes his impression of the object.
For purposes of this project, the field team members were
instructed to use a scale of one to eight in making their assess-
ments. A value of "one" was assigned to the most favorable overall
conditions and a value of "eight" to the worst overall conditions.
Intermediate points were used to reflect conditions in between the
two values.
The data gatherers were instructed to ignore those factors
not related to the study in making these ratings (e. g., the color
of the house, the socio-demographic characteristics of the neigh-
borhood, and so forth). They were requested to form their overall
opinion solely on the basis of the data they were collecting on the
individual effectiveness measures.
Assessment of the Relationship Among Variables
The relationship between the global ratings and the corre-
sponding component variables was examined in light of the three
decision models. The models were tested using regression
analysis techniques. The mathematical expressions used to
formulate the models were of the type illustrated in Appendix A.
A mathematical approach was utilized in preference to a
judgmental approach for the following reasons:
The difficulties associated with implementing judg-
mental techniques when a sizeable number of vari-
ables is to be assessed.
46
-------
The difficulties associated with converting qualitative
statements into statistical terms when more than one
judge (or evaluator) is involved.
The opinions of the officials in the health, sanitation, and
planning departments in the pilot test city were, however, solicited
in conjunction with another aspect of the study. A summary of the
results of this survey is presented in Appendix B.
In examining the relationship between the composite rating
and the component variables, the data collected in the one census
tract that all observers visited were used. The data from this
tract were pooled across all raters; that is, simple average values
were computed for the global ratings and for the component vari-
ables. The resultant equations were compared in terms of their
correlation coefficients, and index values were developed, using
the regression parameters as weights.
In addition, the relationship between the overall rating and
the component variables was tested separately for selected raters
(data gatherers). This was performed in order to compare the
types of decision models the various raters used with the models
that resulted from the pooled data.
47
-------
V. THE FIELD DEMONSTRATION
This chapter describes the methods that were used to assess
the measurement system, particularly its implementation in an
urban community. The City of Baltimore was used as the test
site for the field demonstration.
The field demonstration consisted of:
An evaluation of the usefulness of the various candidate
effectiveness measures by potential users of the mea-
surement system.
A field test of the measurement techniques in selected
areas of the city.
To assess the usefulness of the candidate effectiveness mea-
sures, the list of measures, illustrated in Table 3 on pages 30
through 36, was presented to a group of city government repre-
sentatives in the test city. The group included representatives
from the sanitation, health, and planning departments. They were
asked to separately assess each variable in terms of how useful
it would be to them in their decision-making needs. The results
of this survey are summarized in Appendix B.
To assess the measurement techniques, a field test was per-
formed, during which data were collected in ten areas of the city.
It lasted for a two-week period and included a three-day training
session in which ten persons were instructed in how to make the
measurements and record the resultant data on a set of data col-
lection forms.
The remainder of this chapter provides additional information
on the field test. It covers the following points:
The purpose of the field test
The principal focus of the field test
The survey design for the field test
49
-------
The activities performed in preparation for the field
test
The conduct of the field test.
PURPOSE OF THE FIELD TEST
The purpose of the field test was to obtain information by
which to evaluate:
• The feasibility of producing the individual and com-
posite measures (in terms of time, cost, and diffi-
culty involved).
• The consistency (or reliability) with which subjective
measurements are made; i. e., the extent to which
independent observers come up with similar measure-
ments of the same phenomena.
• The extent to which different measures are likely to
be highly correlated with one another; i. e., if clean
alleys are highly correlated with clean storage areas,
it may not be necessary to measure both variables in
future studies of this type.
• The degree to which individual variables vary in dif-
ferent sections of the city.
• The feasibility of developing composite effectiveness
indices.
FOCUS OF THE FIELD TEST
As illustrated in Table 3 on pages 30 through 36, four types
of data were used to formulate the candidate effectiveness mea-
sures. Some measures utilized existing data records and ledgers;
for example, the number of trash fires per 1,000 persons, the
number of complaints about schedule delays per 1,000 persons,
and so forth. Some required the use of special instruments; for
example, the noise from collection trucks. A third category of
measures was that which required a household survey; for example
an index of citizens' attitudes about the appearance of streets and
50
-------
alleys in their community. The majority of the measures, how-
ever, were of a nature that required direct observation of existing
conditions; for example, the number of storage areas containing
health hazards.
The field test focussed on the latter type of measures. It
was originally envisioned that data from records and ledgers
would also be included. This did not prove feasible for the follow-
ing reasons:
It would have required manual tabulation based on
source materials, many of which were not centrally
located.
The method for recording complaint data in the pilot
test city was such that real complaints were often
indistinguishable from "requests for service. "
The sanitation department itself was in the process of
changing over to a borough system, thereby making
data on collection routes and collection schedules
difficult to obtain.
Because of these factors, an attempt to formulate measures based
on recorded data would have required designing a management in-
formation system that, on one hand, may have only limited appli-
cability and, on the other hand, was beyond the scope of the project.
The four measures that required the use of special measure-
ment apparatus — rat counts, fly counts, noise, and vehicle pol-
lution— were excluded because of feedback received from city
government representatives in the test city during the design phase
of the study. These officials were of the opinion that measurement
of flies and rats using special measuring devices was no more
reliable or accurate than similar measurements made visually by
trained inspectors.'" It was, therefore, decided to adopt the latter
These comments are not based on scientific experiments
designed to compare the various measurement techniques;
rather ^ they reflect the professional opinion of several
management officials in the health department of the test city.
51
-------
technique in making these types of measurements. Noise and
vehicle pollution were not directly measured because the city
government managers, in their evaluation of the effectiveness
measures, rated them low in terms of usefulness. Additionally,
it was felt that specialized measures of these types might be too
cumbersome and costly for an urban community to adopt on an
ongoing basis.
A Citizen Attitude Survey was not undertaken because this
technique has been tested on a number of occasions and found to
be feasible, if properly designed and if the city is willing to commit
the needed resources. A fair amount of work in this area has been
done by the Urban Institute, Washington, D. C.14
SURVEY DESIGN FOR THE FIELD TEST
Having defined the scope of the pilot test as one that would
concentrate on measures that require visual inspection of exist-
ing conditions, a sampling plan was developed. The sampling
plan was based on the concept of sampling from among and within
strata. Each stratum represented a set of census tracts, having
certain things in common. Since no census tract was split between
strata, each stratum represented a unique set of census tracts.
There are a number of alternative schemes by which the
strata could be defined. The scheme used for the field test de-
fined five strata as follows:
• Dirty Stratum—Those census tracts the sanitation and
health department personnel defined as particularly
dirty.
• Model Cities Stratum — Those census tracts contained
within the Model Cities areas of the city.
• Income Stratum No. 1 — Those census tracts where the
average family income in 1970 was less than $9,000.
• Income Stratum No. 2—Those census tracts where the
average family income in 1970 was between $9,000 and
$11,999.
• Income Stratum No. 3 — Those census tracts where the
average family income in 1970 was $12,000 or more.
52
-------
To eliminate any overlap among strata, census tracts belonging
to the Dirty Stratum were classified first, followed by tracts be-
longing to the Model Cities Stratum. The group of census tracts
that remained were then classified into one of the three income-
related strata.
The Dirty Stratum was included to ensure that a maximum
amount of variation would be observed in the individual measures.
The Model Cities Stratum was included because this area was
receiving more frequent service (with respect to regular and bulk
collection and street and alley cleaning) than the rest of the city.
Using information available from the 1970 Census of Housing,
two census tracts were randomly selected from each stratum, and
ten census blocks were selected at random from each tract.15 Thus,
altogether 100 blocks were chosen for observation during the field
test.
The sampling plan called for ten observers to perform the
data gathering. They were to form five teams of two observers
each. Each team member was to record his (or her) observations
separately. The 100 blocks were divided up among the five teams
so that:
• All observer teams inspected all census tracts, but
surveyed different blocks within the tracts.
• Each census tract was inspected on each day of the
week.
Thus, the basic survey design consisted of a set of interpenetrating
samples, the details of which are more fully described in Appendix C.
The basic survey design was augmented by having all observ-
ers inspect all blocks in one tract, termed the "common tract. "
These inspections were made daily, at approximately the same
time, by all teams. The intensive inspection of one census tract
was added to the basic survey design in order to facilitate the sub-
sequent analysis of the measurement methodology, particularly the
reliability and accuracy of the measurements and the development
of effectiveness indices.
53
-------
ACTIVITIES PREPARATORY TO THE FIELD TEST
Several activities were performed prior to the actual con-
duct of the field test to facilitate the collection of data. These
included:
• Conducting a pre-survey of the units to be sampled
• Designing data collection forms and procedures
• Developing training aids and other related materials.
Each is briefly described below.
Pre-Survey of the Census Blocks in the Sample
Members of the project team visited each block that was to
be surveyed during the field test. This was done for two reasons:
To prepare block maps that could be used by the field
personnel
To confirm the fact that a selected block actually con-
tained homes.
The block maps were developed in order to assist the field
personnel in locating the correct block and in recording data. The
maps indicated the shape of each block, the street names, and the
location of the alleys. The streets and alleys were subsequently
assigned numeric identifiers that were used by the observers when
recording information. A replica of a block map is shown on the
following page in Figure 3.
Several of the selected census tracts were in areas where
urban renewal was underway. In some instances, all of the homes
on a given block had been torn down (or condemned) since the 1970
census was taken. Where this was the case, the block was removed
from the sample and a replacement block from the same census
tract was selected.
54
-------
Latona
Goodwood Rd.
Figure 3: Replica of a Block Map That Was Prepared for the
Field Test
Development of Data Collection Forms and Procedures
A set of five data collection forms was developed to facili-
tate the recording and subsequent computer processing of the
data. These forms, replicas of which are provided in Appendix
D, were designed to capture information for those measures
noted in Table 3 as requiring direct observation of conditions.
A detailed description of the data collection procedures was pre-
pared in the form of an instruction booklet that was given to all
field observers. Appendix D provides a summary of these pro-
cedures.
Development of Training Materials and Other Aids
A training program was developed to familiarize the field
personnel with the measurement system concepts, the data col-
lection forms, and the field test procedures. The training ma-
terials consisted of:
55
-------
• An instruction booklet explaining the recording pro-
cedures for each of five data collection forms
• Opaque projector slides
• Color transparencies (slides)
• A filmstrip
• Maps
• Handouts.
CONDUCT OF THE FIELD TEST
The field test was conducted during the first two weeks in
December. It included a three-day training session in which the
ten participants received instructions on how to locate the sample
blocks, make the required measurements, and complete the data
collection forms. The remainder of the time was spent collecting
the data from the ten census tracts that were to be surveyed.
The Training Session
The training session included both structured and situational
type instruction. That is, visual aids and other materials were
used to explain what was meant by each item on the data collection
forms and how the various items should be completed. Special
attention was given to clarifying the subjective-type measurements
that required use of rating scales; namely, measurements of the
amount of glass, garbage, and refuse. This was all done in a
classroom-type setting.
The field observers were then split up into small groups,
sent out to a sample block, and asked to complete the data forms
by themselves. Each group was accompanied by a member of the
project team, who answered questions about the recording pro-
cedures. The field observers, however, were requested to seek
answers for themselves, using their instruction booklets, prior
to asking the project analyst.
56
-------
Following the on-site training, the group was reassembled.
They were asked to rate the conditions that were presented in a
series of slides. The slides focussed on situations where the
recording procedure required the use of a rating scale. After
each slide, the assigned ratings were discussed and clarified. >r<
The Field Procedures
Subsequent to the training session, the ten observers were
split into five teams of two people each, to begin the actual col-
lection of data. Each person was asked to complete the data col-
lection forms by himself (or herself), independent of the other
team member. That is, they were to refrain from discussing
what they recorded before, during, or after completion of the
forms.
The field personnel assembled each morning and were
assigned from four to six new blocks to survey that day, including
several blocks in the "common" tract where all raters went each
day. Using the census tract and block maps that were provided,
field team members drove to the sample block, parked their cars,
and conducted the inspection on foot.
For each block, they collected data on the following six
observational areas:
Blockfaces — The area from the center of the street
up to and including the curb and gutter, extending
from any corner of a street to an adjacent corner.
Private Ways — Sidewalks and front yards bordering
blockfaces.
Alleys — Passageways, usually 5 to 10 feet wide,
extending into or through the interior of a block.
It was as a result of these discussions that the initial scale,
which was a four-point scale, was revised to a seven-point
scale to allow for intermediate measurement points.
57
-------
Backyards—Areas in the rear of a structure, bounded
by the property lines of adjacent structures, and, in
most cases, by an alley.
Storage Areas—Areas (external to the structure) that
normally serve as the location for the refuse containers.
Lots and Public Places—Open areas, having no struc-
tures.
Figure 4 on the following page provides a graphical display of
these six areas.
Blockfaces, alleys, and private ways were inspected in seg-
ments of approximately 100 feet. Information was recorded sepa-
rately for each segment. Team members were asked to mutually
agree as to the specific boundaries of the area being included in
each segment, to ensure the comparability of the responses of the
team members. Approximately eight storage and backyard areas
were inspected in each block. The field personnel were given in-
structions on how to randomly select the areas for inspection.
All lots in the block were inspected.
Types of Information Collected and Recording Procedures
The types of information that were collected for each of the
six observational areas listed above are summarized in tabular
form on page 60. The table also indicates the recording method
associated with each measurement element.
Essentially, there were four major types of recording pro-
cedures:
Seven-Point Rating Scales—These were used to rate
the amount of garbage, glass, and refuse observed in
each of the six observational areas. The following
codes were to be assigned:
1 = None observed
3 = Minor amount observed
58
-------
L
n i r
LEGEND:
) Blockfaces
IOOOOI Private Ways
l Backyards
IOOOI Storage Areas
r
Figure 4: A Sketch of the Six Observational Areas
That Were Inspected in Each Survey Block
59
-------
Table 4. SUMMARY OF THE INFORMATION
COLLECTED AND THE RECORDING PROCEDURES
BY TYPE OF MEASUREMENT
Type of
Measurement
Health
Hazards
Safety
Hazards
Unsightly
Conditions
Odor
Satisfaction
of Needs
Compliance
with
Standards
Overall
Assessment
Information Collected
Garbage Rating
Rat Indicators
Insect Indicators
Dead Animals
Glass Rating
Fire Hazards
Refrigerator With Door
Refuse Rating
Discarded Bulk Items
Abandoned Vehicles
Clogged Drain Basins
Odor
Containers
Improper Containers
Non- complying Containers
Composite Rating
Observational
Areaa
all
all
Storage areas
all
all
all
all
all
aU
all
Blockfaces, alleys
Storage areas
Storage areas
Storage areas
all
Recording Method
7-point scale
Yes/no
Yes/no
Number observed
7-point scale
None /minor /major
Yes/no
7-point scale
Number observed
Number observed
Number observed
Yes/no
Number observed
(by size)
Yes/no
Number observed
8-point scale
This refers to the six areas for which data were collected: blockfaces, alleys,
private ways, backyards, storage areas, and lots.
60
-------
5 = Moderate to heavy amount observed
7 = Substantial amount observed
Intermediate values of 2, 4, and 6 were used when in
between conditions were observed.
Yes/No Codes — These indicated the presence or
absence of a given condition.
Counts — These reflected the number observed of a
given item.
Composite Rating Scale — This scale, ranging from
1 to 8, was used by the observer to indicate his (or
her) subjective evaluation of the conditions observed.
Code 1 was used to indicate the most favorable overall
condition. Code 8 was used to indicate the worst
overall condition. The scale progressed at one-point
intervals to indicate conditions in between the two
extremes.
In addition, there was one measurement that utilized a recording
procedure other than those listed above; namely fire hazards.
This was recorded utilizing one of the following three descriptors:
none, minor, or major.
61
-------
VI. FINDINGS AND CONCLUSIONS
This chapter presents the major findings and conclusions
that were developed based on the field demonstration. These
findings are of two types:
• General findings
• Findings related to the sample design.
GENERAL FINDINGS
The general findings of this project may be summarized as
follows:
• The number of variables measured can be reduced
because many of the variables were frequently found
to be at their lowest value.
• Only three observational areas need to be inspected —
blockfaces, alleys, and lots.
• For blockface measurements, one blockface selected
at random can be used in lieu of all four blockfaces
to measure the effectiveness variables.
• Observers exhibit a high degree of consistency for
the "yes-no" type measurements and for "counts. "
• Observers exhibit a fair degree of consistency for the
more subjective-type measurements; i.e., glass,
garbage, and refuse ratings. The amount of variation
is less at lower scale points than at higher scale points,
tending to rapidly increase and then stabilize.
• Variation among observers, however, only accounts
for approximately 15 to 20 percent of total variation
when measuring a tract mean.
63
-------
On the whole, observers tend to be accurate to within
one-half of a scale point for the glass, garbage, and
refuse ratings; a few, however, were always high or
always low in their assigned ratings.
There are a number of statistically significant corre-
lations among the variables; however, the explained
variation tends to be low.
It is possible to develop composite measures of effec-
tiveness, using multivariate techniques; however, the
refuse rating by itself can serve as a proxy for an
overall measure.
These findings are discussed in more detail in the paragraphs
that follow. The analysis plan on which many of the findings are
based is provided in Appendix E. Detailed tables, charts, and
graphs that support the findings are presented in Appendix F.
The Number of Variables Measured Can Be Reduced
When a frequency distribution of each of the measurements
was reviewed, it became apparent that many of the variables were
frequently at their lowest value. By lowest value is meant the
absence of the condition being measured. In the case of the rating
scales, the lowest value was 1, while for counts, it was 0.
Table 5 on the following page illustrates this point. It shows
for each variable the percent of time when the lowest value did not
occur. The table indicates that only the following variables are
likely to exhibit variation:
Refuse Rating
Glass Rating
Garbage Rating
Rat Indicators in Alleys, Storage Areas, Backyards,
and Lots
Bulk Items in Alleys, Storage Areas, Backyards, and
Lots.
64
-------
Table 5. DISTRIBUTIONAL CHARACTERISTICS
OF THE VARIABLES
Variable
Refuse Rating
Glass Rating
Garbage Rating
Rat Indicators in Alleys, Storage
Areas, Backyards, and Lots
Bulk Items in Alleys, Storage
Areas, Backyards, and Lots
Clogged Drain Basins
Odors
Rat Indicators in Blockfaces and
Private Ways
Bulk Items in Blockfaces and
Private Ways
Fire Hazards
Insects
Abandoned Vehicles
Dead Animals
Refrigerator With Door
% of Observations Where
Lowest Value Did Not Occur
All Tracts in
Sample a
65
45
19
17
15
5
4
4
2
2
1
1
1
0
Common Tract
Only b
89
85
37
71
28
6
c
11
3
2
c
0
1
0
a
b
This is based on several thousand observations.
This is based on slightly less than a thousand observations and
reflects data on blockfaces, private ways, and alleys only.
Data not available.
65
-------
This would tend to suggest that only these variables need to be
measured.
For comparative purposes, the table presents information
for the common tract by itself (i. e. , the tract where all observers
went) as well as for all tracts combined. The common tract was
the tract that exhibited the worst conditions overall. Thus, it is
interesting to note that the same five variables listed above were
also the predominant ones in this tract as well in terms of the
frequency with which they exceeded their minimum levels.
More detailed information on the distribution of the measure-
ments is provided in Appendix F, Tables F-l through F-14 and
Figures F-l and F-2. These tables and figures present the fre-
quency of occurrence of each variable for all the sample tracts
combined and for the common tract by itself.
Only Three Observational Areas Need to Be Inspected — Blockfaces,
Alleys, and Lots
There is a sufficient degree of correlation among similar
measurements when compared across the six observational areas
to suggest that data need be collected only for the following obser-
vational areas:
Blockfaces
Alleys
Lots.
Conditions observed along private ways were found to be
related to conditions observed along blockfaces. Shown at the top
of the following page are the correlation coefficients and explained
variation for the three measurements that exhibited the most sen-
sitivity; namely, refuse, glass, and garbage ratings.
66
-------
Blockfaces and Private Ways
Variable
Refuse Rating
Glass Rating
Garbage Rating
Coefficient of
Correlation
(R)
.94
.96
.80
Explained
Variation
(R2)
.88
.92
.64
Conditions along alleys were found to be related to conditions
observed in storage areas and backyards. The correlation co-
efficients and the explained variation between the refuse, glass,
and garbage ratings for alleys and storage areas, and for alleys
and backyards, are as follows:
Alleys and Storage Areas
Variable
Refuse Rating
Glass Rating
Garbage Rating
Coefficient of
Correlation
(R)
.72
.79
.66
Explained
Variation
(R2)
. 52
.63
.44
Alleys and Backyards
Variable
Refuse Rating
Glass Rating
Garbage Rating
Coefficient of
Correlation
(R)
.84
.84
.88
Explained
Variation
(R2)
.71
.71
.78
67
-------
It was difficult to assess the degree of correlation between
lots and other observational areas, because of the small number
of lots in the total sample. Inspection of the data, however, re-
veals that lot characteristics do tend to differ from those of the
other five observational areas.
For Blockface Measurements, One Blockface Selected at Random
Can Be Used in Lieu of All Four Blockfaces to Measure the
Effectiveness Variables
During the pilot test, observers collected data on all four
sides of each block that they inspected. The findings indicate
that this level of detail is generally not needed to accurately esti-
mate a tract mean rating for refuse, glass, or garbage. Rather,
it would be sufficient to inspect only one blockface of each block
in the sample.
In deriving this conclusion, mean refuse, glass, and garbage
ratings were computed for each of the ten census tracts. One set
of ratings was based on all four blockfaces in each block that was
sampled. Another set of ratings was based on only one randomly
selected blockface in each of the blocks. The mean values were
compared using a t-test. The results, shown in Tables 6 through
8, indicate that, in general, there is no statistically significant
difference in the mean values computed using the two methods.
Observers Exhibit a High Degree of Consistency for the "Yes-No"
Type Measurements and for "Counts"
The raters were in agreement 96 percent of the time or
better for those measurements that required either an assessment
of the presence or absence of a given condition or a count of the
number of similar type items that were present. Table 9 on page
72 shows the extent of agreement by variable for these types of
measurements.
The item of particular significance on this table is the level
of agreement among observers when making measurements of rat
indicators, since this particular variable is probably the most
subjective of those on the list, and, hence, likely to contain higher
amounts of rater variation.
68
-------
Table 6. BLOCKFACE MEAN GARBAGE RATINGS FOR
CENSUS TRACTS — RATINGS BASED ON ALL BLOCKFACES
IN A BLOCK VERSUS RATINGS BASED ON ONE RANDOMLY
SELECTED BLOCKFACE PER BLOCK
\ Mean Garbage Rating
\ Based on All
\ Blockfaces in
CensusX the Sample
Tracts \ Blocks
1
2
3
4
5
6
7
8
9
10
1.43
1.67
1.02
1.15
1.06
1.67
1.04
1.05
1.23
1.30
Based on One
Randomly Selected
Blockface in Each
Sampled Block
1.25
1.91
1.00
1.13
1.05
2.22
1.05
1.07
1.41
1.11
Absolute Value of
the Difference
Between the
Two Ratings
.18
.24
.02
.02
.01
.55
.01
.02
.18
.19
Significant Difference
Established at
90%
Confidence
Level
No
No
No
No
No
Yes
No
No
No
Yes
95%
Confidence
Level
No
No
No
No
No
No
No
No
No
Yes
69
-------
Table 7. BLOCKFACE MEAN GLASS RATINGS FOR
CENSUS TRACTS — RATINGS BASED ON ALL BLOCKFACES
IN A BLOCK VERSUS RATINGS BASED ON ONE RANDOMLY
SELECTED BLOCKFACE PER BLOCK
\ Mean Glass Rating
\ Based on All
\ Bio ckf aces in
CensusN. the Sample
Tracts \ Blocks
1
0
fj
3
4
5
6
7
8
9
10
2.53
3.17
1.55
2.28
1.62
2.93
1.71
1.31
2.01
3.81
Based on One
Randomly Selected
Bio ckf ace in Each
Sampled Block
2.14
3.29
1.35
2.20
1.35
3.09
1.91
1.33
1.82
4.17
Absolute Value of
the Difference
Between the
Two Ratings
.39
.12
.20
.08
.27
.16
.20
.02
.19
.36
Significant Difference
Established at
90%
Confidence
Level
Yes
No
No
No
Yes
No
No
No
No
No
95%
Confidence
Level
No
No
No
No
No
No
No
No
No
No
70
-------
Table 8. BLOCKFACE MEAN REFUSE RATINGS FOR
CENSUS TRACTS — RATINGS BASED ON ALL BLOCKFACES
IN A BLOCK VERSUS RATINGS BASED ON ONE RANDOMLY
SELECTED BLOCKFACE PER BLOCK
\ Mean Refuse Rating
\ Based on All
\ Blockfaces in
CensusX the Sample
Tracts \ Blocks
1
2
3
4
5
6
7
8
9
10
3.01
3.43
1.43
2.85
2.13
3.23
2.36
1.47
3.11
3.91
Based on One
Randomly Selected
Blockface in Each
Sampled Block
2.75
3.90
1.53
3.93
1.65
3.91
2.64
1.33
3.35
3.94
Absolute Value of
the Difference
Between the
Two Ratings
.26
.47
.10
1.08
.48
.68
.28
.14
.24
.03
Significant Difference
Established at
90%
Confidence
Level
No
No
No
Yes
Yes
Yes
No
No
No
No
95%
Confidence
Level
No
No
No
Yes
Yes
No
No
No
No
No
71
-------
Table 9. RATER AGREEMENT FOR MEASUREMENTS
OTHER THAN RATING SCALES
Recording
Method
Yes-No
Number
Observed
None / Minor /Major
Variable
Rat Indicators
Odors
Insects
Refrigerator With Door
Bulk Items
Clogged Drain Basins
Abandoned Vehicles
Dead Animals
Fire Hazards
% of Observations
Where Observers
Agreed
96.3
97.7
99.8
100.0
96.9
97. 7
99.7
99.8
98.0
These results should be viewed with some caution, however.
This is because many of the variables listed in Table 9 were among
those found to exhibit little variation. That is, many of them did not
frequently deviate from their lowest value. (See Table 5 on page 65.)
Observers Exhibit a Fair Degree of Consistency with Respect to
the More Subjective-Type Measurements—Namely, the Glass.
Garbage, and Refuse Ratings
It is to be expected that measurements based on the three
rating scales, because of their subjective nature, are likely to
exhibit a higher degree of inconsistency than would be the case
with other types of measurements made during the field test.
The field test data indicated that, on the average, the ob-
servers differed by less than one point from one another on a seven-
point scale, with 1 representing the most favorable conditions and
7 representing the most unfavorable conditions. The average
72
-------
difference across all six observational areas between pairs of
observers for the garbage, glass, and refuse ratings was found
to be .32, .65, and .82 points, respectively. This information is
summarized in the table below, which also indicates the frequency
with which raters agreed within one point or less of one another
as well as the percent of observations where observers agreed
within two points or less of one another.
Table 10. RATER AGREEMENT FOR
RATING SCALE MEASUREMENTS
Type of
Rating Scale
Garbage
Glass
Refuse
% of Observations
Where Raters
Agreed Within
1 Point or Less
89.1
79.1
73.7
% of Observations
Raters Agreed
Within 2 Points
or Less
98.4
94.6
93.7
Mean
Rating
Difference
.32
.65
.82
The difference between observers in their assigned ratings
was also analyzed to see whether the size of the rating discrepancy
was related to the scale points. The initial hypothesis that ob-
servers would tend to show fairly close agreement for points 1
and 7 (the scale extremes) and much less agreement for the middle
points of the scale did not prove out. Rather, inspection of a
graphical plot of the rating discrepancy by scale point revealed
that the amount of variation between raters showed a tendency to
increase rapidly at the lower scale points and then stabilize. That
is, the close agreement that was expected at the upper end of the
scale did not materialize.
This relationship between the scale points and the mean dif-
ferences was estimated statistically for each of the three rating
scales, using the following function form:
Y = a - b/X
73
-------
where X = the rating scale point
Y = the mean difference associated with the given
scale point.
The following regression equations were developed, each of which
was found to provide a fairly good fit to the data:
Type of 2
Rating Scale Estimating Equation R_
Garbage Y = 1.704 - 1.636/X .895
Glass Y = 1.465 - 1.226/X .884
Refuse Y = 1.358 - 0.967/X .896
Figures 5 through 1, shown on the following three pages,
present graphically the rater variation by scale point for the gar-
bage, glass, and refuse rating scales, respectively. These figures
show the actual average difference between observer pairs at each
scale point and the 95 percent confidence interval that is associated
with the respective mean differences. In cases where there were
a sizeable number of observations (over a hundred or so) for a
given scale point, the confidence bands are fairly tight. However,
where the observations are few in number, the interval about the
mean difference tends to be much larger. The figures also show
the estimating relationships that were developed.
Appendix F, Tables F-15 through F-17 present estimates of
the variance and standard deviation and the numeric values asso-
ciated with the 95 percent confidence interval about the mean dif-
ferences contained in Figures 5 through 7. A description of the
methodology used in developing these findings is provided in
Appendix E.
74
-------
LU
o
z
UJ
CC
01
Li.
U.
Q
a
UJ
0
CO
cc
C3
z
<
2.00
1.50
1.00
0.50
Total No. of
Comparisons:
% of Total:
Y=1.704-1.636/X
1.58
1.40
1.10 -L
1.18
1.59 •
Average Rating = 1.43
Average Difference Between
Observer Pairs = 0.32
1234567
RATING
5052 329 598 30 154 11 66
80.9% 5.2% 9.6% 0.5% 2.5% 0.2% 1.1%
Figure 5: Mean Garbage Rating Difference by Scale Point
(including 95% confidence interval about the mean
differences)
75
-------
2.00
ID
o
z
LU
cc
UJ
1.50
1.00
CC
w
c/j
<
UJ
0.50
1.10
Average Rating = 2.31
Average Difference Between
Observer Pairs = 0.65
0
(
Total No. of
Comparisons:
% of Total:
| |
) 1 2
3410 479
54.7% 7.7%
| |
345
RATING
1224 158 481
19.6% 2.5% 7.7%
6
65
1.1%
7
419
6.7%
Figure 6: Mean Glass Rating Difference by Scale Point
(including 95% confidence interval about the mean
differences)
76
-------
LU
o
2
LLI
DC
LU
Q
0
LU
CO
D
LU
LLI
DC
LU
2.00
1.50
1.00
0.50
.42
.87
Average Rating = 2.89
Average Difference Between
Observer Pairs = 0.82
0-
0
Total No. of
Comparisons:
% of Total:
1 234567
RATING
2162 710 1684 273 731 71 605
34.7% 11.4% 27.
0% 4.4% 11.7% 1.1% 9.7%
Figure 7: Mean Refuse Rating Difference by Scale Point
(including 95% confidence interval about the mean
differences)
77
-------
Variation Among Observers Accounts for Approximately 15 to 20
Percent of the Total Variation When Measuring a Tract Mean
In making statistical inferences about a given geographical
area, there are a number of sources of variability that can effect
the estimate. Some of the major components that contribute to the
variance of a census tract mean glass, garbage, or refuse rating
are the following:
• Variation among blocks in the tract
• Variation among observers
• Random sources of variation.
When these variance components were estimated, by apply-
ing analysis of variance techniques to the data that were collected
on the common tract, it was discovered that the major source of
variation stemmed from the block to block differences. This
accounted for approximately 75 percent of the total variation in
the sample. The observers, on the other hand, contributed only
15 to 20 percent. Random effects made up the balance of the
total variation. Both the block effect and the rater effect were
significant at the .001 level. Table 11 on the following page
summarizes the findings with respect to these three sources of
variation in the average refuse and glass ratings for blockfaces
and alleys in the common tract.
These findings provide insight into the relevance (or prac-
tical importance) of the fact that observers do not always agree
on the rating to be assigned to a particular condition. In addition,
the findings indicate that if objective measurements were to be
utilized, the variance about the mean could be reduced, but the
real key to reducing the variance about a tract mean that is esti-
mated from sample data is the number of blocks in the sample.
The methodology used in developing these conclusions is
described in Appendix E.
-------
Table 11. THE VARIANCE COMPONENTS FOR A CENSUS
TRACT MEAN VALUE FOR GLASS AND REFUSE RATINGS
IN BLOCKFACES AND ALLEYS
(expressed in percentage terms)
Type of Rating Scale
Glass Rating (Blockfaces)
Glass Rating (Alleys)
Refuse Rating (Blockfaces)
Refuse Rating (Alleys)
% Contribution of Variance
Components
Blocks
76
78
73
76
Raters
18
15
20
18
Random Effects
6
7
7
6
On the Whole, Observers Tend to Be Fairly Accurate in Their
Assessment of Garbage, Glass, and Refuse Conditions; A Few,
However, Were Always High or Always Low in Their Assigned
Ratings
In general, the observers were able to estimate the average
tract ratings for a census tract to within +_ one-half of a scale
point. This is illustrated more clearly in Table 12 on the follow-
ing page. This table indicates the amount by which each observer
differed from the overall tract value for selected rating scales.
Data from the common tract were used in this regard. The distri-
bution of the rater means about the overall tract mean is displayed
graphically in Appendix F, Figures F-3 through F-8.
Table 12 suggests that a few of the observers tended to
systematically assign substantially higher or lower ratings than
the rest of the group. In particular, observer 8 appears to con-
sistently assign ratings that exceed the mean by approximately
one scale point, while observer 10 tends to always be on the low
side of the overall tract average by about one scale point. This
observation was borne out when the blockface mean glass and refuse
ratings of the observers were compared against the overall mean
ratings by block for the ten blocks that comprise the common tract.
79
-------
The results are shown in Figure 8 which is presented on five
separate pages, starting on page 81. The graphs contained in
this figure illustrate that:
Some of the raters were inherently more variable
than the rest; i.e., some tended to show larger
fluctuations about the block means.
Observer 8 systematically gave higher ratings to
each block in the common tract.
Table 12. DIFFERENCES BETWEEN THE AVERAGE RATER VALUES
AND THE OVERALL AVERAGE VALUES OF THE GARBAGE,
GLASS, AND REFUSE RATINGS FOR BLOCKFACES AND
ALLEYS IN THE COMMON TRACT
Observer
1
2
3
4
5
6
7
8
9
10
Overall
Tract
Mean
Difference Between Average Rater Values
and the "True" Average Values a
Blockface
Garbage
Rating
.11
.02
-.02
.08
.44
-.30
.01
.34
-.37
-.09
1.42
Blockface
Glass
Rating
.26
-.27
.10
.27
-.42
-.05
.90
1.22
-.33
-. 97
3.42
Blockface
Refuse
Rating
.38
-.13
-.34
.21
-.63
-.39
.49
1.68
.24
-.94
3.51
Alley
Garbage
Rating
.12
-.33
.45
-.42
1.71
1.02
-.42
.41
-1.08
-.67
3.15
Alley
Glass
Rating
.13
-.32
.11
-.31
-.09
.19
1.07
1.33
.27
-1.95
5.02
Alley
Refuse
Rating
-.01
-.91
-.04
.03
.18
.45
.81
1.22
-.09
-1.14
5.31
The "true" average values reflect the overall tract means
all ten observers.
across
80
-------
OBSERVER 1 COMPARED WITH BLOCK AVERAGES
OC
UJ
s
u.
UJ
oc
UJ
5
I
I
I
I
I
> 6
CC
UJ
V)
CO _
O 5
C3
Z
I -
UJ
I
I
_1_
I
J
1234567
MEAN REFUSE RATING - ALL OBSERVERS
1234567
MEAN GLASS RATING - ALL OBSERVERS
(M
OC
UI
I 6
UJ
I 4
u.
UJ
OC
z
UJ
5
OBSERVER 2 COMPARED WITH BLOCK AVERAGES
CM
OC
UJ
> 6
OC
UJ
i <
I
5 4
I
I
I
I
1
I
1234 567
MEAN REFUSE RATING - ALL OBSERVERS
1234567
MEAN GLASS RATING - ALL OBSERVERS
Figure 8: Blockface Glass and Refuse Ratings of Observers Compared
With Overall Ratings for Blocks in Common Tract
-------
OBSERVER 3 COMPARED WITH BLOCK AVERAGES
co 7
cc
UJ
K 6
LU
P 4
<
IT
UJ
H
<
UJ
I
I
J_
1234 567
MEAN REFUSE RATING - ALL OBSERVERS
oc
UJ
> 6
cc
UJ
§.
I
(3
1 4
1
I
I
I
1234 567
MEAN GLASS RATING - ALL OBSERVERS
OBSERVER 4 COMPARED WITH BLOCK AVERAGES
IT
UJ
UJ
CO
ffi
O
P 4
Ul
s 3
_L
_L
_L
J
1234 567
MEAN REFUSE RATING - AL.. OBSERVERS
> 6
cc
Ul
i -
i
o
* 4
z
UJ
_L
I
I
I
1234 567
MEAN GLASS RATINb - ALL OBSERVERS
Figure 8 (continued)
82
-------
OBSERVER 5 COMPARED WITH BLOCK AVERAGES
10
cc
LJJ
O
4
u.
UJ
e ,
z
UJ
2 1
I
I
1234 567
MEAN REFUSE RATING - ALL OBSERVERS
in
CC
UJ
> 6
CC
111
CO
00
O 5
I
? 4
I
<
UJ
I
I
_|_
J
1234 567
MEAN GLASS RATING - ALL OBSERVERS
OBSERVER 6 COMPARED WITH BLOCK AVERAGES
OC
UJ
4
ui ,
co 3
u.
UJ
I *
<
ui
I
I
I
I
1234 567
MEAN REFUSE RATING - ALL OBSERVERS
a:
UJ
> 6
DC
UI
to
CO
O 5
2 4
CC
to 3
eo
i ^
<
LU
_L
_L
1234567
MEAN GLASS RATING - ALL OBSERVERS
Figure 8 (continued)
-------
OBSERVER 7 COMPARED WITH BLOCK AVERAGES
71-
cc
LU
oo
CO c
O 5
I
Z 4
LL.
UJ
UJ
2
_L
_L
J
1234 567
MEAN REFUSE RATING - ALL OBSERVERS
r-
yj 6
>
oc
UJ
CO c
co 5
O
I
U
_|_
I
I
_J_
1234 567
MEAN GLASS RATING - ALL OBSERVERS
OBSERVER 8 COMPARED WITH BLOCK AVERAGES
OC
UJ e
> 6
cc
UJ
CO
m _
O 5
I
4
u.
UJ
cc 2
1
I
I
I
I
J
123 4567
MEAN REFUSE RATING - ALL OBSERVERS
00
OC
OC
UJ
O
I
<
=
Z
UJ
S
_L
_|_
I
I
J
1234 567
MEAN GLASS RATING - ALL OBSERVERS
Figure 8 (continued)
84
-------
OBSERVER 9 COMPARED WITH BLOCK AVERAGES
o>
cc
3! 6
tc.
Ul
m _
O 5
I
oc
C/J
CC 2
z
<
LJ
S 1
_L
I
I
I
1234 567
MEAN REFUSE RATING - ALL OBSERVERS
o>
ID 6
>
ec
UJ
8 5
o
i
UJ
5
1234 567
MEAN GLASS RATING - ALL OBSERVERS
OBSERVER 10 COMPARED WITH BLOCK AVERAGES
> 6
OC
UJ
CO
CD _
O 5
I
O
J_
1234 567
MEAN REFUSE RATING - ALL OBSERVERS
uj
>
cc
UJ
CO
m
O
I
1234 567
MEAN GLASS RATING - ALL OBSERVERS
Figure 8 (continued)
85
-------
Observer 10 systematically assigned lower values to
each block in the common tract.
Although Figure 8 indicates that observers 5 and 7 also tend to be
consistently different, similar patterns for these two observers
were not found to hold true for some of the other rating scales.
Observers 8 and 10, on the other hand, were systematically dif-
ferent across all rating scales.
There Are a Number of Statistically Significant Correlations
Among the Variables; However, the Explained Variation Tends
to Be Low
Correlation analysis was used to test the degree to which
individual pairs of variables were related with one another. These
tests were conducted on the set of variables used to assess alley
conditions as well as on the set of variables used to measure
blockface conditions.
The results indicated that statistically significant correla-
tions exist among pairs of variables. That is, there is a high
probability that the real association between the two variables is
not zero. However, as illustrated in Tables 13 and 14, shown on
the following page, the correlation coefficients between pairs
of variables are not high. Thus, although statistically meaningful
relationships do exist, they do not explain a sufficient amount of
the variation to be considered as good predictors of each other.
This implies that if one is interested in knowing the amount of
garbage, a separate measurement of garbage conditions will be
required.
The correlations shown in Tables 13 and 14 are based on
data from the common tract. This tract was used because the
individual variables showed more sensitivity in the common tract
than they did in any of the other tracts.
86
-------
Table 13. CORRELATIONS AMONG THE VARIABLES
USED TO MEASURE BLOCKFACE CONDITIONS
Variable Pairs
Refuse and Glass Ratings
Refuse and Garbage Ratings
Glass and Garbage Ratings
Correlation
Coefficient
(R)
.590
.207
.164
Amount of
Explained
Variation
(R2)
.348
.043
.027
Level of
Significance
.001
.001
.001
Table 14. CORRELATIONS AMONG THE VARIABLES
USED TO MEASURE ALLEY CONDITIONS
Variable Pairs
Refuse & Glass Ratings
Refuse & Garbage Ratings
Rat Indicators & Garbage Rating
Rat Indicators & Refuse Rating
Glass & Garbage Ratings
Rat Indicators & Glass Rating
Bulk Items & Refuse Rating
Fire Hazards & Garbage Rating
Fire Hazards & Refuse Rating
Bulk Items & Glass Rating
Correlation
Coefficient
(R)
. 597
.473
.365
.346
.308
.287
.261
.244
.202
.173
Amount of
Explained
Variation
(R2)
.356
.223
.133
.120
.095
.082
.068
.060
.041
.030
Level of
Significance
.001
.001
.001
.001
.001
.001
.001
.001
.001
.001
87
-------
It is Possible to Develop Composite Measures of Effectiveness
Using Multivariate Techniques
Overall measures of effectiveness were developed for block-
faces and alleys, based on data obtained from inspections made in
the common tract. These particular observational areas were
focussed upon because of a prior finding indicating that conditions
in these two areas are related to conditions in three of the other
observational areas in the sample. Indices for lot conditions were
not developed because of the small number of observations obtained
for lots. The methods used to develop the indices are described in
more detail in Appendix E.
Blockface Indices
The functional relationships associated with the linear;
conjunctive, and disjunctive models were estimated using
regression techniques. The following least-squares equations
were developed:
Type of
Model
Linear
Conjunctive
Disjunctive
A
InY,
Estimating Equation
= 8.236 -.448X
1B
-.289X
2B
.3451n(8-X1TJ +.2141n(8 -XotJ
1 ri 6X5
In Y0= 1.960 - .1691n(X, - .93) - .1101n(XOT, - .93)
O.D 1.O £>l5
R
.883
.879
.865
where
X
X
IB
2B
JB
value of the blockface refuse rating
value of the blockface glass rating
estimated value for the blockface
overall effectiveness rating.
In each case, only the coefficients associated with the refuse
and glass ratings proved to be significant at statistically
acceptable levels.
88
-------
These equations were developed by averaging the values
that individual observers assigned to blockfaces in the common
tract. Similar functional relationships were also estimated
for selected observers. However, in these cases the linear
and conjunctive models were clearly favored over the disjunc-
tive model. (See Appendix F, Tables F-18 through F-22. )
Of interest in comparing the three equations is that all
assign approximately the same relative weights to the inde-
pendent variables. This may be more clearly seen when the
estimating parameters are normalized so that their sum adds
up to one, as illustrated below.
Relative Relative
Type of Weight for Xj Weight for X2
Model (Refuse Rating) (Glass Rating)
Linear .608 .392
Conjunctive .617 .383
Disjunctive .606 .394
For comparative purposes, the three regression
equations are illustrated graphically on the following two
pages in Figures 9 and 10. Figure 9 presents the relation-
ship between refuse rating values and estimated values of
the overall effectiveness rating when the glass rating is held
constant at various levels. Figure 10 illustrates the relation-
ship between glass rating values and estimated values of the
overall effectiveness rating when the refuse rating is held
constant at various levels.
It should be pointed out that at certain values for the
refuse and the glass ratings, the graphs indicate large dis-
crepancies among the three curves, discrepancies much
larger than would be expected based on the R2 values shown
above. This is explained by the fact that the data used to fit
the three equations contained relatively few of the extreme
values. * The largest discrepancies among the three equations
This is due primarily to the fact that averages across ob-
servers were used. Any averaging technique tends to obscure
the extreme values.
89
-------
15
C3 13
CO
to
O
01
LL
LL
LU
CC
LU
O
11
1
Glass Rating
(X2) =
01234567
REFUSE RATING
O
z
CO
CO
LU
H
LU
O
LU
O
15
13
11
1
Glass Rating (X2) = 3
01234567
REFUSE RATING
CD
Z
LU
Z
LU
O
LU
U.
LL
LU
CC
LU
O
1
Glass Rating
(X2)
01234567
REFUSE RATING
O
Z
CO
CO
O
LU
LU
CC
LU
O
1
Glass Rating (X2) = 7
0123456
REFUSE RATING
Curve A: Disjunctive Model
Curve 6: Linear Model
Curve C: Conjunctive Model
Figure 9: Graphical Presentation of the Regression Equations
Developed for Blockface Data: Relationship Between
Refuse Rating and Overall Effectiveness Rating When
Glass Rating is Held Constant
90
-------
o
2
LU
Z
LU
P
O
cc
UJ
o
Y
15 -
13 -
11
1
Refuse Rating
01 234567
GLASS RATING
UJ
z
UJ
o
LU
U.
LL
UJ
DC
LU
O
15
13
11
1
Refuse Rating (X,) = 3
01234567
GLASS RATING
U)
GO
LU
z
UJ
O
LL
LL
LU
cc
LU
o
Refuse Rating
(X ) = 5
123456
GLASS RATING
O
z
CO
UJ
z
o
LU
LU
_J
CC
LU
O
Refuse Rating (X.,) = 7
123456
GLASS RATING
Curve A: Disjunctive Model
Curve B: Linear Model
Curve C: Conjunctive Model
X2
Figure 10: Graphical Presentation of the Regression Equations
Developed for Blockface Data: Relationship Between
Glass Rating and Overall Effectiveness Rating When
Refuse Rating is Held Constant
91
-------
occur when one or both of the independent variables is shown
to have an extreme value. Thus, in the relevant range for
which the data were fit, all three curves may be said to be
fairly close to one another.
In the case of the three equations presented above, the
coefficient of determination (R2) provides little guidance for
selecting a decision model that might underlie an overall
effectiveness measure. However, the disjunctive model is
probably a poor choice for two reasons. First, it takes on
most of its driving force when the independent variables are
at their extreme values. However, as mentioned above, the
data used to fit the model contained relatively few extreme
values, leading one to suspect that the R2 value may be arti-
ficially inflated. Second, when the model was tested using
individual data for selected observers, where extreme values
were included, the fit to the data was not so good. Rather,
observers tended to exhibit linear or conjunctive type com-
bining tendencies.
The three equations were transformed to indices,
having scales that range from 0 to 1, where 1 reflects the
preferred value, by use of the following mathematical manipu-
lation:
(A \ /
j j(min)) //.
Y - Y
'/ I j(max) j(min)
A
where Y. = value predicted by the regression
equation (j = 1, 2, 3).
The resultant index formulas for the three models are shown
in Table 15 on the following page.
Thus, it is possible to construct indices for blockface
conditions; however, it should be pointed out that the refuse
rating by itself can serve as a good proxy for an overall
measure. When considered by itself, the refuse rating ex-
plained 83 percent of the variation in the case of the linear
and conjunctive models presented above and 80 percent of the
variation in the disjunctive model. Relatively high correlations
92
-------
between the refuse rating and the overall rating were
also found in the estimating relationships developed for
individual observers.
Table 15. INDEX FORMULAS
FOR BLOCKFACE CONDITIONS
Type of Model
Linear
Conjunctive
Disjunctive
Index Formula
A
I1T, = .226 Y113 - .
IB IB
A
I = . 208 Y0 - .
OT2 OT3
-------
where
X
X
X
1A
2A
3A
Y.
value of the alley refuse rating
value of the alley glass rating
value of the alley garbage rating
estimated value for the alley overall
effectiveness rating.
In both cases, the coefficients associated with the refuse,
glass, and garbage ratings were statistically significant
at the .001 level.
Unlike the estimating equations for blockfaces, these
equations do not assign the same relative weights to the in-
dependent variables. As shown below, when the estimating
parameters are normalized so that their sum adds up to one,
the linear model places heaviest weight on the refuse rating,
while the conjunctive model places heaviest weight on the
garbage rating:
Relative
Type of Weight for Xj
Model (Refuse Rating)
Linear .493
Conjunctive .338
Relative
Weight for X£
(Glass Rating)
.216
.197
Relative
Weight for Xs
(Garbage Rating)
.291
.465
Index formulas, scaled from 0 to 1, for these two
models are provided in Table 16 below. Here also, however,
Table 16. INDEX FORMULAS
FOR ALLEY CONDITIONS
Type of Model
Index Formula
Linear
Conjunctive
- .279
- .236
94
-------
the refuse rating by itself can serve as a good proxy for
an overall measure. When considered by itself, the alley
refuse rating accounted for 91 percent of the variation in
the case of the linear model and 86 percent of the variation
in the case of the conjunctive model.
FINDINGS RELATED TO THE SAMPLE DESIGN
In addition to the general findings described above, there
were several findings that related to the sample design. They may
be summarized as follows:
Large differences in the mean tract ratings were found
to exist between two groupings of the strata.
The sampling plan was adequate to detect changes of
one-half a point or less in the blockface ratings; larger
samples would be required to detect an equivalent
change in the alley ratings.
The mean tract ratings did not appear to be biased by
the day of the week when the inspection was made.
These findings are discussed in the paragraphs that follow.
Large Discrepancies in the Mean Tract Ratings Were Found to
Exist Between Two Groupings of the Strata
As described in Chapter V of this report, the sampling plan
was based on the concept of strata; i.e., mutually exclusive sets
of census tracts. The five strata used in the project were: Dirty,
Model Cities, Income Level 1, Income Level 2, and Income Level 3.
When the mean garbage, glass, and refuse values for each
stratum were compared with those of the other strata, two distinct
groups appeared to emerge, consisting of the following strata:
95
-------
Group 1
Group 2
Dirty
Model Cities
Income Level 1
Income Level 2
Income Level 3
This two-fold split is illustrated in Table 17, shown below, which
presents the mean blockface and alley ratings for each stratum.
Table 17. AVERAGE REFUSE, GLASS, AND
GARBAGE RATINGS FOR BLOCKFACES
AND ALLEYS BY STRATUM
Stratum
Dirty
Model Cities
Income Level 1
Income Level 2
Income Level 3
Blockface Mean Ratings
Refuse
Rating
3.55
2. 95
3.27
1. 95
1.78
Glass
Rating
3.42
2.43
2.59
1.53
1.39
Garbage
Rating
1.47
1.32
1.45
1.05
1.04
Alley Mean Ratings
Refuse
Rating
4.90
4.23
4.36
3.28
2. 58
Glass
Rating
5.17
4.17
4.74
2. 53
1.93
Garbage
Rating
2. 88
1.91
2. 23
1.44
1.26
Significance tests on the difference between the mean values
confirmed that, in general, when a mean rating for a stratum from
Group 1 was compared with a mean rating for a stratum from
Group 2, the two ratings were found to be different at statistically
significant levels. On the other hand, when the difference between
the mean ratings for two strata within the same group was tested,
no statistically significant difference between the two means could
be established. *
The t-test was used to determine whether or not the mean
values between stratum were significantly different.
96
-------
In interpreting the results, it should be pointed out that the
Model Cities and Dirty Strata consisted almost entirely of census
tracts where the average annual family income (in 1970) was less
than $9,000. This corresponds to the income range that was used
to define the Income Level 1 Stratum. Thus, Group 1 can be said
to consist of census tracts where the average annual family income
is less than $9,000, while Group 2 contains those census tracts
where the average annual family income is $9,000 or more. Inso-
far as the mean ratings for these two groups were found to be sub-
stantially different, with Group 1 having much higher ratings than
Group 2, the results tend to support a hypothesis that many have
postulated; namely, the amount of glass, garbage, and refuse
found in a given area is related to the income level of the area.
The Sampling Plan Was Adequate to Detect Changes of One-Half a
Point or Less in the Blockface Ratings
In general, the higher the tract mean rating, the larger the
standard error. In spite of this, the sampling plan proved ade-
quate to detect a change of one-half a point or less in the average
tract values for blockface ratings.
As illustrated in Table 18 on the following page, the highest
average refuse rating for blockfaces in the ten census tracts
sampled was 3.91. Given its standard error of the mean of .242,
a change of + .49 points or more is discernible. Where the mean
refuse ratings were lower, the standard errors are smaller and,
therefore smaller changes in the mean ratings are detectable.
Similar results were obtained when the average garbage and
glass ratings for blockfaces were analyzed; i. e., the sampling plan
enabled one to detect changes of one-half a point or less in the
average values for a census tract. The refuse rating has been used
(in Table 18) to illustrate the point because it generally had larger
standard errors and slightly higher mean values than the other two
blockface rating scales.
Since higher mean ratings and larger variances tend to be
associated with census tracts where the average annual family
income is less than $9,000, this finding implies that more inten-
sive sampling is required in these areas to detect a given amount
of change in the blockface ratings than is needed in census tracts
where the income level is $9,000 or more.
97
-------
Table 18. AMOUNT OF CHANGE IN THE CENSUS
TRACT MEAN BLOCKFACE REFUSE RATINGS
DETECTABLE AT THE 95 PERCENT CONFIDENCE LEVEL
(based on sample design for field test)
Census Tract
Mean Refuse
Ratings
3.91
3.43
3.23
3.11
3.01
2.85
2.36
2.13
1.47
1.43
Standard
Error of the Mean
(<7_)
v x'
.242
.202
.233
.168
.186
.187
.141
.127
.097
.085
Change in Rating
Detectable at
95% Level of Confidence
.49
.41
.47
.34
.38
.38
.28
.26
.20
.17
For alley ratings, on the other hand, the sampling plan
proved inadequate in all but a few cases to detect a change of one-
half a point or less in the mean tract ratings. Table 19 on the
following page summarizes the mean alley refuse ratings for the
ten census tracts sampled, the standard error associated with
each mean, and the amount of change that would be detectable.
The results imply that a larger number of alleys would need to be
sampled if one wants to be able to detect small changes in the
alley ratings.
98
-------
Table 19. AMOUNT OF CHANGE IN THE CENSUS
TRACT MEAN ALLEY REFUSE RATINGS
DETECTABLE AT THE 95 PERCENT CONFIDENCE LEVEL
(based on sample design for field test)
Census Tract
Mean Refuse
Ratings
5.38
4.46
4.40
4.30
4.27
4.15
3.61
3.30
1.78
1.50
Standard
Error of the Mean
(a—)
v X;
.306
.336
.359
.267
.360
.384
.321
.337
.152
.139
Change in Rating
Detectable at
95% Level of Confidence
.64
.70
.75
.56
.75
.80
.67
.70
.32
.29
The Mean Tract Ratings Did Not Appear to Be Biased by the Day
of the Week When the Inspection Was Made
Analysis of variance tests were conducted using the data
collected by several of the observers to see if there were any
statistically significant day-to-day changes in the ratings in the
common tract. As mentioned previously, the common tract was
the area that exhibited the worst conditions overall. The three
observers for whom the analysis was carried out were among the
three most accurate and most conscientious raters.
The results tend to suggest that in areas where the mean
ratings are high, inspections can be made on any day of the week
and still be representative. In only a few of the cases tested was
there a significant difference or bias in the ratings due to the day
of the week.
99
-------
VII. RECOMMENDATIONS
The recommendations stemming from this project are
of two types:
General recommendations on how to develop a
measurement system that is specific to a given
community.
Detailed recommendations on how to implement an
ongoing measurement system using the findings
from the field test data.
Each is briefly described below.
RECOMMENDATIONS RELATED TO DEVELOPING A MEA-
SUREMENT SYSTEM
The following recommendations are offered for commu-
nities who want to use the general procedures that were
developed during this project as an aid in designing a solid
waste effectiveness measurement system that is unique to
their own community:
(1) Review the list of measures and measurement
techniques (provided in Table 3 on pages 30
through 36) and select those measures most
useful.
(2) Use the basic survey design developed for this
project to obtain preliminary data on those
measures that require direct observation of
existing conditions. Income ranges appear to be
the most viable candidate for establishing sam-
pling strata.
101
-------
(3) Utilize several observers in each tract and have
them make the same measurements.
(4) Apply techniques similar to those used in this
project to determine the appropriate sample
size for an ongoing measurement system and the
relevant variables for which measurements should
be made.
RECOMMENDATIONS RELATED TO IMPLEMENTING AN
ONGOING MEASUREMENT SYSTEM
The recommendations presented below are offered to
communities who want to implement an effectiveness measure-
ment system that is based upon the results of the field demon-
stration conducted in the City of Baltimore.
(1) Collect data on blockfaces, alleys, and lots only.
(2) Sample approximately ten blocks in each census
tract where the overall conditions are bad to
detect a change of one-half a point or less in the
blockface garbage, glass, and refuse ratings;
inspect fewer blocks in areas where the overall
conditions are good. Use the income level of the
census tract as an initial means for classifying
conditions.
(3) Inspect only one randomly selected blockface in each
block; inspect all alleys and lots in the block.
(4) Utilize several observers and have them inspect
different blocks in the same census tract in order
to reduce the variation associated with inconsistency
among observers.
(5) Periodically compare the observations of raters
within the same tract to see if any of the observers
are consistently high or consistently low.
102
-------
(6) Make measurements only of the amount of refuse
found in these areas. Use this as an indicator of
overall conditions.
or
Make measurements of the amount of refuse, glass,
and garbage found in the areas, the presence of
rat signs (alleys only) and the number of bulk items
(alleys only). Report the measurements separately
and/or as a composite measure.
(7) Report the location of fire hazards, bulk items,
abandoned vehicles, clogged basins, and other
items of interest so that corrective action can be
taken.
103
-------
CITED REFERENCES
1. Blair, L. H., and A. I. Schwartz. How Clean Is Our City?
Washington, The Urban Institute, 1972.
2. "Community Litter Survey Technique. " New York, Keep
America Beautiful, Inc.,, May 5, 1973. (Working Paper.)
3. Ralph Stone and Company, Inc. The Use of Bags for Solid
Waste Storage and Collection. U.S. Environmental Pro-
tection Agency, 1972. [Distributed by National Technical
Information Service, Springfield, Va., as PB-212 590.] p. 21.
4. Fund for the City of New York. Unpublished data, 1973.
5. D. C. Department of Environmental Services. Unpublished
data.
6. "Sanitation Management Information System Concept. "
CONSAD Corporation, January 1972. (Unpublished Report.)
7. "Effectiveness Report on the Use of Plastic Bags as Trash
Receptacles in the City of Baltimore. " Baltimore, Depart-
ment of Public Works, September 24, 1973. (Working
Paper.)
8. Einhorn, H. J. "The Use of Nonlinear, Noncompensatory
Models in Decision Making. " Psychological Bulletin,
73 (3): 221-230, 1970.
9. Klee, A. J. "The Role of Decision Models in the Evaluation
of Competing Environmental Health Alternatives. " Manage-
ment Science, 18(2): B-52 to B-67, October 1971.
10. Helmer, O. "Analysis of the Future: The Delphi Method. "
Defense Doc. Center Document AD 649640. The Rand
Corporation, 1967. pp 1-11.
11. Baumol, W. J. Economic Theory and Operations Analysis.
2d ed. New Jersey, Prentice-Hall, Inc., 1965. pp 355-385.
105
-------
12. Arrow, K. J. Social Choice and Individual Values, Cowles
Commission Monograph No. 12. New York, John Wiley
and Sons, Inc., 1951.
13. Klee, A. J. "The Utilization of Expert Opinion in Decision-
Making." AlChE Journal, 18(6): 1107-1115, November
1972.
14. Webb, K., and H. P. Hatry. Obtaining Citizen Feedback;
An Application of Citizen Surveys to Local Governments.
Washington, The Urban Institute, 1973.
15. U.S. Bureau of the Census. 1970 Census of Housing;
Block Statistics. Final Report HC(3)-106, Baltimore,
Maryland Urbanized Area. Washington, U.S. Govern-
ment Printing Office, 1971.
106
-------
APPENDIX A
FURTHER DISCUSSION OF THE
LINEAR. CONJUNCTIVE, AND
DISJUNCTIVE DECISION MODELS
This appendix presents mathematical representations
for the linear, conjunctive, and disjunctive decision models that
were used in the project. In addition, geometric illustrations
are provided for the latter two models.
THE LINEAR MODEL
The linear model produces an index that is an additive sum
of the component variables. For example, if health, safety, and
appearance are the principal indicators in assessing solid waste
system effectiveness, the values of the variables associated with
these indicators are weighted to reflect their relative importance
and then added together to obtain an overall score. This score is
the index value for the given values of the variables.
The linear model may be represented by the following gen-
eral form:
n
E = £ w x
where E represents an overall (or global) assessment of conditions.
The wi are the weights that are given to the individual component
variables, x..
THE CONJUNCTIVE MODEL
The conjunctive model says that the effectiveness of a solid
waste system depends on whether each of the component variables
used to measure effectiveness surpasses a threshold value set for
it. If E = f (x,, x2, . . . , xn) represents the vector of effectiveness
variables, and T = g (t,, t2, . . ., tn) represents the vector of thresh-
olds, then only if x^ is greater than ti for all i will the system be
107
-------
effective. Since a certain minimum value is required for all the
variables, this implies that a high score on one variable cannot
compensate for a low score on another variable, as is the case
with the linear model. Thus, the conjunctive model adheres to
a multiple cutoff procedure rather than a linear compensatory
procedure.
In the strict sense in which the model is defined, the function
is a discontinuous one that can take on the values of "l" or "0"
only. The value of the function is "l" whenever all the x^ are
greater than the corresponding tj_; otherwise the value is "0. "
A geometric representation of this model is shown in Figure A-l(a)
on the following page. There are a number of mathematical forms
that are continuous in nature that can be used to approximate the
conjunctive model. In this project, the following parametric function
was used:
n
w.
x. i
i
where the variables are defined as in the linear model.
A geometric representation of this function is provided in
Figure A-l(b). The nature of this function is such that a low value
for any one of the component variables will produce a low overall
value. Moreover, the low value cannot be compensated for by
high scores on the other variables, since it is the product of the
variables that is the important factor.
To facilitate the use of ordinary least square methods in
fitting the data to the model, a linear transformation of the above
function can be performed by taking the logarithms of both sides
of the equation. The model thus becomes:
n
In E = Y] w.ln(x.)
108
-------
*E = 1
i
-
X, threshold
Conjunctiva E = 1 if X, > X, threshold and X2>X2 threshold
E = 0 otherwise
(a)
*X, .Y^ (E = 1)
/ • •
threshold-
(b)
X, threshold
Continuous, differentiate approx. to a conjunctive model
Figure A-l: Geometric Representations
of the Conjunctive Model
109
-------
Because the raw data to be used in fitting the least squares
function were scaled such that the smaller the value the better
the conditions, the variables were converted to a reverse scale.
This had the effect of producing a function that retained the prop-
erties of the one being discussed but was its mirror image.
THE DISJUNCTIVE MODEL
This disjunctive model says that the effectiveness of a solid
waste system depends on whether at least one of the component
variables used to measure effectiveness exceeds the individual
threshold set for it. If E = f (x,, x2, .. ., x^ represents the
vector of effectiveness variables, and S = h (s,, s2, • •., sn)
represents the vector of thresholds, then if x^ exceeds s^ for at
least one i, the system is effective. * Since only one component
variable needs to exceed its standard, the disjunctive model is
called a maximum evaluation function, as compared with the con-
junctive model which is a minimum evaluation function.
In the strict sense in which the model is defined, the function
is a discontinuous one that can take on the value's of "l" or "0"
only. The value of the function is "l" whenever any one of the x^
exceed the corresponding s^; otherwise the function has a value
of "0. " A geometric representation of this model is presented in
Figure A-2(a), shown on the following page.
As in the case of the conjunctive model, there are a number
of mathematical forms that can be used to approximate the disjunc-
tive model and give it a continuous shape. A hyperbolic function,
having the following form, was used in this project:
Si is used here as a threshold symbol instead of t^ to indicate
that the threshold values for the disjunctive model may be
different from those for the conjunctive model.
110
-------
X. ,X- (E = 6) Xi threshold
'b *b
Disjunctive
E • 0 if X, < X, threshold andX^ X, threshold or X2 > X^ threshold
threshold
X1 threshold
Continuous, differentiate approx. to a disjunctive model within
a region of desired application.
Figure A-2: Geometric Representations
of the Disjunctive Model
(a)
(b)
111
-------
where the p^ are the asymptotic values of the function. They are
arbitrarily set for each x^, such that they exceed the maximum
value that the x^ can attain.
The function is illustrated geometrically in Figure A-2(b).
The nature of this function is such that a high value for any one of
the component variables will produce a high value for the overall
measure.
Performing a linear transformation, the function becomes:
n
InE = -£ w. In (p. - x.)
X l
As in the case of the conjunctive model, the raw data were con-
verted to a reverse scale, and the estimated function was a mirror
image of the one being discussed.
112
-------
APPENDIX B
AN ASSESSMENT OF THE USEFULNESS
OF THE CANDIDATE EFFECTIVENESS MEASURES
This appendix presents the results of a survey in which a
group of officials in the pilot test city were asked to assess the
usefulness of the effectiveness measures, illustrated in Table 3
on pages 30 through 36. The group of evaluators consisted of
representatives from the sanitation, health, and planning depart-
ments of that city. The evaluators separately assessed each
variable in terms of how useful it would be to them in their de-
cision-making needs. They were asked to assign one of the
following three descriptors to each variable:
important
of limited value
not important.
The variables were then scored as follows, based on the descrip-
tors assigned:
2 = important
1 = of limited value
0 - not important.
Mean scores were computed by major measurement cate-
gory (health, appearance, safety, etc. ) and by solid waste activity
(storage, collection, etc. ). These are shown for all evaluators
combined in Table B-l on the following page.
The results indicate that in terms of solid waste activities,
the most useful variables are those related to:
Storage
Collection
Cleaning.
113
-------
Table B-l. VARIABLE SCORES FOR THE CANDIDATE
EFFECTIVENESS MEASURES BY SOLID WASTE
ACTIVITY AND BY MEASUREMENT CATEGORY
(Averages for all Evaluators Combined)
^S. Solid Waste Activity
Measurements. Storage
Category ^S^
Public Health
Public Safety
Appearance of
Community
Odor
Satisfaction of
Storage and
Collection Needs
Compliance with
Standards
Inconvenience to
Public
All Categories
Combined
1.6
1.5
1.8
0.8
1.6
1.6
1.6
Collection
1.7
1.6
2.0
0.7
1.6
1.4
1.1
1.4
Cleaning
1.8
1.7
1.6
1.6
Local
Trans-
portation
1.2
0. 8
0.9
All
Activities
Combined3
1.7
1.6
1.7
0.7
1.6
1.5
1.0
1.4
Weighted averages.
114
-------
In terms of measurement categories, the most useful variables
are those related to:
• Public Health
• Public Safety
• Appearance of Community
• Satisfaction of Community's Storage and Collection
Needs
• Compliance with Standards.
Table B-2 on pages 116 through 120 presents the score for
each effectiveness measure by measurement category. It reflects
the average score across all evaluators.
115
-------
Table B-2. VARIABLE SCORES FOR EACH OF THE
CANDIDATE EFFECTIVENESS MEASURES
BY MEASUREMENT CATEGORY
(Averages for all Evaluators Combined)
Measurement
Category
Measures of Effectiveness
Variable
Score
ffi
H
J
-------
Table B-2 (continued)
Measurement
Category
Measures of Effectiveness
Variable
Score
TJ
0)
fl
o
u
U
a
Percent of storage areas found to contain
safety hazards
Average safety hazard rating for storage areas
Percent of storage areas where safety hazard
rating exceeds a given threshold
Percent of blocks where more than "X" percent
of the storage areas is found to contain safety
hazards
Percent of inspections found to contain safety
hazards on the public way
Percent of inspections where safety hazards
are found on lots or public areas
1.8
1.6
1.6
1.8
1.6
O
U
fc
O
w
u
PH
Average appearance rating for storage areas
Percent of storage areas where appearance
rating exceeds a given threshold
Percent of storage areas found to contain
abandoned or discarded bulk items
Percent of blockfaces containing spilled or
scattered refuse subsequent to collection (where
curbside collection is performed)
Percent of alleys containing spilled or scattered
refuse subsequent to collection (where alley
collection is performed)
Percent of inspections where abandoned or
discarded bulk items are observed
— Percent of inspections where abandoned auto-
mobiles or trucks are observed
117
1.8
1.8
2.0
2.0
2.0
-------
Table B-2 (continued)
Measurement
Category
Measures of Effectiveness
Variable
Score
!*
o .
u
fin
Average litter rating for streets and alleys
Percent of streets and alleys where litter rating
exceeds a given threshold
Number of unsolicited complaints from citizens
about the appearance of their community per
1, 000 persons
Index of citizen satisfaction with the appearance
of their community
Average litter rating for vacant lots and public
areas
Number of drain basins which are clogged per
basin inspected
Percent of collection fleet likely to cause
spillage while in transport
1.8
1.2
1.2
1.6
2.0
1.2
Percent of storage areas found to contain
offensive odors
Percent of blocks where more than "X" percent
of the storage areas is found to contain
offensive odors
Number of unsolicited complaints from citizens
about the existence of offensive odors per
1, 000 persons
0.8
0.8
0.6
s 5
<1 o ^ H
fo H H EJ
02 to J G
2 O s
-------
Table B-2 (continued)
Measurement
Category
SATISFACTION
OF STORAGE
& COLLECTION
NEEDS
(Continued)
COMPLIANCE WITH STANDARDS
INCONVE-
NIENCE/
DISCOMFORT
TO PUBLIC
Measures of Effectiveness
— Percent of storage areas found to contain an
inadequate number of containers
— Percent of blocks where more than "X" percent
of the storage areas contains an inadequate
number of containers
— Percent of storage areas having containers
which do not comply with the regulations
— Percent of blockfaces where mixed refuse re-
mains uncollected for one or more days
(where curbside collection is performed)
+
Percent of alleys where mixed refuse remains
uncollected for one or more days (where alley
collection is performed)
— Average delay time in meeting regular pickup
schedules for alleys /blockfaces
— Number of unsolicited complaints from citizens
about delays in pickup of mixed refuse per
1, 000 persons
— Percent of instances in which there was a delay
of one day or more in the pickup of bulky items
— Average delay time in the pickup of bulky items
— Number of unsolicited complaints from citizens
about delays in pickup of special or bulk items
per 1, 000 persons
— Average amount of time spent per household
per month preparing refuse for collection
— Percent of collection areas where noise from
collection exceeds a given threshold
Variable
Score
1.8
1.8
1.6
1.8
1.4
1.6
1.6
1.0
1.2
0.6
1.0
119
-------
Table B-2 (continued)
Measurement
Category
Measures of Effectiveness
Variable
Score
U
PQ
PH
O
EH
P5
O
Eq _
||
Bl
^o
U
w
I
O
U
Number of collection miles taking place during
early morning hours as a percent of total
collection miles
Number of collection stops taking place during
early morning hours as a percent of total
collection stops
Number of unsolicited complaints from citizens
about noise caused by refuse collection activi-
ties per 1, 000 persons
Number of miles of major and secondary
arterial roads where refuse collection is per-
formed during peak hours
Amount of peak hour time during which refuse
collection is taking place along major and
secondary arterial roads
Number of reported instances of property
damage caused by collection equipment or
collection personnel per 1,000 persons
Total dollar value of property losses caused by
collection equipment or collection personnel
Amount of peak hour time during which
collection fleet is enroute to (or from) central
deposit source
Percent of vehicles where air pollution rating
exceeds a given threshold
1.4
1.2
1.0
1.0
1.2
1.2
1.2
0.4
120
-------
APPENDIX C
DESCRIPTION OF THE SURVEY DESIGN
AND ITS IMPLEMENTATION DURING
THE PILOT TEST
This appendix describes the field survey design and how it
was used during the pilot test phase of the project. The informa-
tion provided in this appendix may be useful to communities in de-
signing their own measurement systems to assess conditions (such
as health hazards, safety hazards, appearance, and so forth) in
various areas such as census tracts, sanitation districts, or entire
cities.
The survey design is based on a concept as to the manner by
which the data at particular observational points are blended togeth-
er to produce estimates for designated areas. This concept enables
one to determine the relative importance and impact of various
sources of variability on the conditions that are being measured.
Some of the sources of variability whose relative importance
should be determined are:
Among days of the week
Among routine observers
Among observation points in the same block
Among blocks in tracts
Among tracts in sanitation districts
Among sanitation districts.
In addition to providing estimates of variation from different
sources, the survey design has a structure that permits unbiased
estimates of the measured conditions to be made for alternative
121
-------
areas. Although the estimates as derived from this survey may
not be sufficiently precise to be of practical value in themselves,
they are sufficient to evaluate the soundness of the estimation
methodology and to determine the size of possible future surveys
so as to achieve a specified level of precision.
GENERAL DESCRIPTION OF THE SURVEY STRUCTURE
The survey structure is based on the concept of sampling
from among and within strata. Strata are defined as mutually
exclusive sets of census tracts; that is, there is no overlap in the
area represented by a given stratum. The primary reason for
formulating strata is to be able to group the sampling units in a
manner that minimizes the sampling variation among units within
a stratum. The reduction of variance within stratum tends to
increase the precision of the estimates.
There are a number of ways to define strata. They could
correspond to different geographical areas, as, for example,
sanitation districts. Alternatively, they could be identified with
income levels.
Once the strata are defined, two census tracts from each
stratum are randomly selected, and ten blocks are selected at
random from each tract. Thus, altogether, ten census tracts
and a total of 100 blocks are covered. The ten blocks of a census
tract are then assigned at random to form five block pairs: 1,
2, 3, 4, and 5. The numbering of the block pairs is independent
from tract to tract; that is, there is no relationship between
block pair j in census tract i and block pair j in some other census
tract. Since there are 10 census tracts and 5 block pairs to each
census tract, there are 50 block pairs.
Ten observers perform the data gathering. They are formed
into five teams of two observers each. The teams are identified as
teams A, B, C, D, and E. With respect to the basic survey design,
a pair of observers inspects a total of ten block pairs. The 50 block
pairs are thus divided into five block pair groups, and each pair of
observers is assigned to a different block pair group.
122
-------
This basic design is augmented in one tract by having all ob-
servers inspect all blocks in that tract. This tract is defined as
the "intensive study" tract. Thus, each observer observes 28 blocks,
composed of two blocks each in nine tracts and ten blocks in the
special intensive study tract.
Because of the weekly mixed refuse collection, estimates of
conditions at a given observation point vary from one day of the week
to another. Consequently, any snapshot for an area should be a
"composite blur" over the conditions of an entire week. For this
reason, the five block pairs of a tract, in addition to being asso-
ciated with different observer pairs, are also associated with
different days of the week. Thus, at the census tract level, effects
of days of the week will be completely confounded with the observer
pair. When data are combined over tracts, it is possible, using
analysis of variance techniques, to separate the effects of pairs
of observers and days of the week.
Table C-l on the following page shows the basic survey design
before randomization. Each letter entry (M, T, W, Th, F) of the
table (indicating the day of the week) refers to the inspection of a
specified block pair by a specified observer pair in a designated
tract. The subscripts of the letters refer to weeks 1 and 2 of the
inspection period. Thus, each inspection team inspects all tracts
and makes inspections on all five days of the week in a balanced
manner. Each tract is inspected on each of the five days of the week
by different observer pairs. Each day of the week is, therefore,
balanced over tracts and observer pairs. In addition to the in-
spections indicated in this table, all observers inspect all blocks
in a selected tract termed the "common" tract. These visits are
all made at approximately the same time by all teams.
Before application of the design set out in Table C-l, various
random selections and randomizations must occur. These are as
follows:
(1) List census tracts in each of five strata and pick two
census tracts at random from each stratum.
(2) Pick 10 blocks at random from each census tract.
123
-------
Table C-l. BASIC SURVEY DESIGN'
Stratum
1
2
3
4
5
Five
Strata
Census
Tract
1
2
1
2
1
2
1
2
1
2
Ten
Census
Tracts
Observer Pair
A B C D
V TI wi Thi
M9 T W Th
6t & & &
TI Wt3 Th1 FI
T W Th F
12 2 2 2
W Th F 5 MI
W Th F 10 M
& & & &
Th F M T
inl 1 1 1
Th F M T
in2 2 2 X2
F M T W
*1 1 1 1
222 2
Number of Block Pairs Per
10 10 10 10
E
Fi
F2
Ml
M2
Tl
T2
Wl
W2
Th 4
2
Number of
Block Pairs
Per Tract
5
5
5
5
5
5
5
5
5
5
Observer Pair
10
The superscripts on the day codes indicate the order of visits to
be made by the person who performed the training. His first in-
spection is indicated by the superscript 1, the second by the super-
script 2, and so on. The observational pattern is balanced over
strata, observer pairs, and days of the weeks. The subscripts
refer to weeks 1 and 2 of the inspection period.
124
-------
(3) Randomly form the 10 blocks within a census tract into
five block pairs. Number these from 1 to 5. Each
block pair should be identified by the stratum (i), the
tract within the stratum (j), and the block pair (k). The
latter are to be numbered from one to five. The blocks
within a pair should be identified by the subscript (m).
The values 1 and 2 taken by the subscript m should be
assigned at random to the two blocks of a pair.
(4) Randomly associate the letters A, B, C, D, and E to
the five block pairs in each tract. There should be a
new randomization for each tract.
(5) Randomly form five pairs of observers from 10 routine
observers. Randomly associate the letters A, B, C,
D, and E to the five pairs of observers. This is done
only once.
(6) Randomly permute the columns of the main body of Table
C-l (that portion that contains the letters indicating days).
After that is done, also randomly permute pairs of rows;
that is, the two rows associated with a stratum should be
treated as one row in the permutation.
SURVEY PLAN AS UTILIZED IN FIELD TEST
The initial survey plan for the field test conducted in the City
of Baltimore utilized the city's five sanitation districts as the strata.
However, when members of the project team surveyed the census
tracts selected from each stratum, they found that there was not
a sufficient amount of variation in the conditions among census
tracts to test the measurement apparatus. For this reason, an
alternative scheme that would ensure an adequate amount of vari-
ation was used for defining the strata. The revised strata were
defined as follows:
Dirty Stratum—Those census tracts the sanitation
and health department personnel defined as particu-
larly dirty.
Model Cities Stratum — Those census tracts contained
within the Model Cities areas of the city.
125
-------
Income Stratum No. 1—Those census tracts where
the average family income in 1970 was less than
$9,000.
Income Stratum No. 2—Those census tracts where
the average family income in 1970 was between $9, 000
and $11, 999.
Income Stratum No. 3—Those census tracts where
the average family income in 1970 was $12, 000 or
more.
To eliminate any overlap among strata, census tracts belonging
to the Dirty Stratum were classified first, followed by tracts be-
longing to the Model Cities Stratum. The group of census tracts
that remained were then classified into one of the three income-
related strata. The total number of census tracts in each stratum
were as follows:
Total
Number of Census
Strata Tracts in Strata
Dirty 15
Model Cities 16
Income Level 1 59
Income Level 2 83
Income Level 3 28
Within each stratum, two census tracts were selected at
random, and from each census tract ten blocks were randomly
selected. The geographical distribution of the census tracts
selected for the field survey within the City of Baltimore is
shown in Figure C-l on the following page. A breakdown of the
survey census tracts by sanitation district and by income group-
ing is provided in Table C-2 on page 128. A similar breakdown
for the entire City of Baltimore is presented in Table C-3 on the
same page for comparative purposes.
126
-------
4 tsoeozl 1507.02 _J 1504 *^
1 '509 U-L—^H
^fflrarSfl
li^^"°ffi£-
1 • n I' \ \ «$/ *\ 2 P\i75y« woNunt«
SlU* \,703>^ 1 1 sp^
jm™{ im\LJr4 501 i
infM 00 CO^^^^^^I X ^i^lV
s|z 0° 5 tuw / wI ^^^^K-' "-^" —
*& g^ „,. ^^^^k BALli^otli_
3 \g "H^^^gzOO^g^M
Ws?lnHS
02 I \ :
tA 2804.03 \'.
2101 kt,,,_,,. ""fe
/ ^"U^lK
/?inu sft so
,2502.05H%I%X^^ ,
^x. \ -«f502-03/^
Figure C-l: Geographical Distribution of Baltimore City Census Tracts
Included in the Field Test
127
-------
Table C-2. DISTRIBUTION OF FIELD TEST CENSUS TRACTS
BY INCOME GROUPING AND SANITATION DISTRICT
""" Income Groupings a
Sanitation
District ^^^
Northeast
Northwest
Eastern
Western
Central
Total
$3,000-
$5,999
2
1
3
$6,000-
$8,999
2
1
3
$9,000-
$11,999
1
1
2
$12,000
and over
1
1
2
Total
2
3
2
3
0
10
a Based on the average family income for 1970. There were no
census tracts where the average family income in 1970 was less
than $3,000.
Table C-3. DISTRIBUTION OF BALTIMORE CITY CENSUS TRACTS
BY INCOME GROUPING AND SANITATION DISTRICT
^*^^^ Income Groupings a
Sanitation
District ^s"\.
Northeast
Northwest
Eastern
Western
Central
Total
$3,000-
$5,999
0
0
8
11
0
19
$6,000-
$8,999
5
15
24
23
0
67
$9,000-
$11,999
25
18
18
26
0
87
$12,000-
and over
9
14
2
2
1
28
Total
39
47
52
62
1
201
Based on the average family income for 1970. There were no
census tracts where the average family income in 1970 was less
than $3,000.
128
-------
APPENDIX D
THE DATA COLLECTION FORMS
AND PROCEDURES
This appendix contains replicas of the data collection forms
that were developed to facilitate the collection of data during the
field test. It includes the following five forms:
Pre-Survey Form — Provides identifying information
on the block to be surveyed.
Blockfaces, Alleys, and Private Ways Form — Pro-
vides information on the conditions observed along
blockfaces, alleys, and front yards and sidewalks.
Storage and Backyard Area Form — Provides infor-
mation on the conditions observed in storage and
backyard areas.
Vacant Lots, Public Parks, and Parking Lots Form-
Provides information on the conditions observed on
lots and parks.
Summary Form — Provides information about collec-
tion and cleaning schedules and the length of time it
took to collect the field data.
The information contained/requested on each of the five forms is
briefly explained in this appendix. Detailed recording procedures
for the forms were provided in the instruction booklet that was de-
veloped for the field test.
THE PRE-SURVEY FORM
The Pre-Survey Form provides information useful in identi-
fying and locating the census block to be inspected. A replica of
this form is provided in Figure D-l on the following page. A
129
-------
PRE-SURVEY FORM
Block Map:
Tract Number
Block Number
Sanitation District
Days of:
regular collection
bulk collection
street cleaning
alley cleaning
Number of:
blockfaces
alleys
open spaces
households
Procedures for Selection of Storage and
Backyard Areas:
Special Notes:
Figure D-l: Replica of the Pre-Survey Form
-------
completed copy of this form was provided for each block that was
surveyed during the field test. It contained:
A hand-drawn map of the block to be surveyed.
A set of code numbers to be used to identify the census
block under inspection.
The various collection and cleaning schedules that
apply to the block.
Some descriptive information about the block.
Procedures to use when selecting storage and back-
yard areas for inspection.
The Block Map, which was hand-drawn on the left-hand side
of the form, contained numeric codes between 1 and 19 to identify
each blockface. These were to be used when recording information
about blockfaces and private ways on the data collection forms.
The-Block Map also contained numeric identifiers for alleys.
These ranged from 21 to 29 and were to be used when recording
information about alleys on the data collection forms.
BLOCKFACES, ALLEYS, AND PRIVATE WAYS FORM
The Blockfaces, Alleys, and Private Ways Data Form is
intended to be used for inspections made in the following areas:
Blockface: The area from the center of the street up to
and including the curb and gutter, extending from any
corner of a block to the adjacent corner.
Alley; A passageway, usually 5 to 10 feet wide, extending
into or through the interior of a block.
Private Way (Sidewalk and Front Yard): The area from
the front of the house to the curb, extending from any
corner of a block to the adjacent corner.
131
-------
A replica of this data form is provided in Figure D-2 on the follow-
ing page.
The above three areas were inspected in linear segments of
approximately 100 to 200 feet each. Data for each blockface and
private way segment were recorded on a separate line on the form.
Similarly, data for each alley segment were recorded on a separate
line. All entries on each line used were to be filled in, unless
otherwise specified. Where one form was not sufficient to record
the information on blockfaces, alleys, and private ways, additional
data forms were used. A separate form was used for each block
that was surveyed.
The form is divided into four major sections:
• Locator Data
• Unit Data
• Blockface and Alley Conditions
• Sidewalk and Front Yard Conditions.
Locator Data is contained at the top of the form in the boxes
corresponding to items 1-29. It provides information by which to
identify the census block being inspected, the date of the inspection,
and the inspection team. Unit data, contained in columns 30-33 of
the form, provides information by which to identify each sample
area inspected. Blockface and Alley Conditions, contained in
columns 34-47^ provides information on the type of conditions ob-
served along each blockface or alley segment inspected. Sidewalk
and Front Yard Conditions, contained in columns 48-59, provides
information on the types of conditions observed along the private
way corresponding to the blockface segment being inspected.
A summary description of the entries on this form (excluding
those connected with the Locator Data at the top of the form) is
provided on pages 134 through 136.
132
-------
Form [Tl Page I I of | |
1 23
Tract No. I I I I l«f"T~l Block No. I I I I Sanitation District No.
4 5 6 7 8 9 10 11 12 13 14
BLOCKFACES, ALLEYS, AND PRIVATE WAYS
Inspector I.D. No. | | |
15 16
Team No. | |
17
Mo. Day Y««r
rn m
18 19 20 21 22 23
Day
24
2526272829
"Descriptions (reference to Blockface or Alley Number, Segment Number, and Column Number):
Figure D-2: Replica of the Blockfaces, Alleys, and Private Ways Data Form
-------
Blockface or Alley Number — The identifying number of the
blockface or alley segment being inspected, as indicated on
the Block Map.
Segment Number — A sequential number assigned by the
observers to each segment.
Percent Residential Code (blockfaces only) — A code number
to describe the percent of residential structures along the
blockface segment being inspected; it was recorded as
follows:
1 0 percent residential
2 = 1-24 percent residential
3 = 25-49 percent residential
4 = 50-74 percent residential
5 = 75-99 percent residential
6 100 percent residential
9 = Not Applicable — an alley is being inspected.
Rat Indicators — Used to indicate the presence of rat signs;
i. e., sighting of live or dead rats, rat gnawings, rat burrows,
rat feces, and rat tracks.
Number of Dead Animals — The number of dead or decaying
animals observed.
Uncontained Garbage Rating — Recorded on a scale of 1 to 7,
with the scale points reflecting varying degrees of uncontained
garbage as shown below:
1 = No uncontained garbage is observed
3 = Minor amounts of uncontained garbage are
observed
5 - Moderate amounts of uncontained garbage are
observed
7 = Substantial amounts of uncontained garbage are
observed or garbage accumulation shows signs
of rat or insect attraction.
Garbage is defined as waste resulting from the preparation,
cooking, serving, or eating of food.
134
-------
Other Health Hazards — Used to indicate the presence of
conditions, not on the data form, that the observer con-
siders to be potential contributors to disease or illness.
Fire Hazard Rating — A code number describing the pres-
ence of fire hazards. It was recorded as follows:
1 = No fire hazards observed
2 = Minor fire hazard exists
3 = Major fire hazard exists.
A major fire hazard is defined as an accumulation of solid
waste materials sufficient to cause or contribute to a fire
of such magnitude that property damage or personal injury
is likely to occur. A minor fire hazard is defined as an
accumulation of solid waste materials sufficient to cause
or contribute to a fire, but unlikely to cause personal injury
or property damage.
Broken Glass/Jagged Objects Rating — Recorded on a scale
of 1 to 7, with the scale points reflecting varying degrees of
glass, jagged objects, etc., as shown below:
1 = No broken glass, jagged objects, etc., are
observed
3 = Minor quantities of broken glass, jagged
objects, etc., are observed
5 = Moderate amounts of broken glass, jagged
objects, etc., are observed
7 - Substantial amounts of broken glass, jagged
objects, etc., are observed.
The items to be included are: broken glass, pieces of barbed
wire, and other sharp or jagged objects.
Refrigerator With Door — Used to indicate the presence of
a refrigerator with a door intact.
Other Safety Hazards — Used to indicate the presence of
conditions, not on the data form, that the observer con-
siders to be potential hazards to public safety.
135
-------
Number of Bulk Items — The number of bulk items observed.
Bulk items are defined as items that cannot fit into a storage
container. These may include discarded furniture items or
appliances, shipping cases, carpeting, automobile tires, and
so forth.
Number of Abandoned Vehicles — The number of abandoned
automobiles and abandoned trucks observed. Abandoned
vehicles are vehicles which appear to be in an apparently
inoperative condition. They are generally characterized by
a lack of licenses or inspection stickers, or by expired
licenses or stickers.
Uncontained Refuse Rating — Recorded on a scale of 1 to 7,
with the scale points reflecting varying degrees of uncontained
refuse as shown below:
No uncontained refuse is lying on the ground
A minor amount of uncontained refuse is
observed
5 = Moderate amounts of uncontained refuse are
observed
7 = Substantial amounts of uncontained refuse are
observed.
The rating should reflect conditions, exclusive of what is
included in making ratings of uncontained garbage, fire
hazards, and broken glass/jagged objects.
Number of Drain Basins (blockfaces and alleys only) —
The total number of storm drain basins observed.
Number of Clogged Drain Basins (blockfaces and alleys
only) — The number of drain basins that appear to be
clogged by debris.
Composite Rating — Recorded on a scale of 1 to 8, with 1
indicating the most favorable overall conditions and 8 in-
dicating the worst overall conditions. This reflects the
observer's subjective assessment of the overall conditions
observed.
136
-------
STORAGE AND BACKYARD AREA FORM
The Storage and Backyard Area Data Form is intended to
be used for inspection of areas where solid waste materials are
stored and for inspection of private areas (backyards) surround-
ing the storage locations. These sample areas are defined as
follows:
Solid Waste Storage Area: An area external to the structure
which normally serves as a location for the containment of
discarded solid waste materials; i. e., the place where
waste storage containers are located.
Private Area (Backyard): The area belonging to the back of
the structure served by the storage area. It is bounded by
the property lines of the adjacent structures and, in many
cases, by an alley. The alley itself, however, is not part
of the inspection area.
A replica of this data form is provided in Figure D-3 on the follow-
ing page.
Only a sample of storage areas and backyards was to be in-
spected within the survey block. Procedures for selection of these
areas were provided on the Pre-Survey Form for the block. Ob-
servers were asked to record data based only on what they observed.
That is, they were not required to open the containers and deter-
mine their contents. In instances where there was no alley access
to the storage/backyard area, the observers were to ask permission
of the resident in order to gain access to these areas. Data for
each storage area/backyard combination was to be recorded on a
separate line on the form.
The format of the Storage and Backyard Area Data Form is
similar to that of the Blockfaces, Alleys, and Private Ways Data
Form. It is divided into the same four major sections and required
many of the same types of measurements to be made.
A summary description of the entries that are unique to this
form is provided on pages 139 through 141.
137
-------
00
Form Q[| PageQ of Q
1 23
Inspector I.D. No. II I
15 16
Tract No. |~| I I !•! ll
4 5 6 7 8 9 10
Block No.n~PI Sanitation District No. D
11 12 13 14
Team No.
17
STORAGE AND BACKYARD AREA
Mo. D«v Ya»r
-mm
18 19 20 21 22 23
2526272829
'Descriptions (reference to Sample Area Number and Column Number):
Figure D-3: Replica of the Storage and Backyard Area Data Form
-------
Sample Area Number — A pre-recorded number that was
used as a reference number for each storage area/back-
yard being inspected.
Blockface Number — The identifying number of the block-
face upon which the structure associated with the sample
area was located.
Structure Code — A code number describing the type of
structure associated with the sample area being inspected;
it was recorded as follows:
1 - Residential
2 = Apartment Complex
3 = Restaurant, Fast Food Establishment
4 = Combination Residential and Business
5 = Business Only
6 = Public Building
7 = Industrial
8 = Other (Specify: )
9 = None.
Collection Responsibility Code — A code number describing
the group responsible for mixed refuse collection; it was
recorded as follows:
1 = Collection performed by city sanitation depart-
ment
2 = Collection performed by private contractor
3 = Unknown (do not know who performs collection).
Number of Regular Collections Per Week — The number of
times per week that mixed refuse is collected, as indicated
on the Pre-Survey Form.
139
-------
Inspection Conditions Code — A code number used to describe
whether there were problems that prevented inspection of
all or part of the designated sample area and, if so, the nature
of the problem. It was recorded as follows:
1 = No problems encountered which hindered the
inspection of either the storage area or the
backyard
2 = No storage area— structure abandoned/unin-
habited
3 = No storage area — structure inhabited
4 = No storage area — unable to determine if
structure is inhabited
5 = Containers are at end of alley or along curb for
pickup
6 = High fence — inspection could not be made
7 = No alley access to storage/backyard area —
resident not at home, or resident would not
allow access
8 = Other (Specify: )
9 = No clearly defined backyard area.
Size of Cans and Bins (storage areas only) — Used to record
the number of storage containers by size and type of container.
Insect Indicators (storage areas only) — Used to indicate the
presence of insect signs; i. e., sighting of swarming or crawl-
ing insects, sighting of insect larvae, and so forth.
Improperly Containerized Garbage (storage areas only) —
A code number to describe the manner in which garbage is
containerized; it was recorded as follows:
1 = All garbage is properly stored in covered cans
or bins
2 = Garbage is lying open in at least one can or bin
and/or garbage is contained in paper sacks,
plastic bags, or other similar containers
3 = There are uncovered cans or bins and/or there
are plastic bags, paper sacks, etc., but it
is not possible to tell whether or not they
contain garbage.
140
-------
A code 2 condition was to be reported in preference to any
other code that may fit the storage area under observation.
Number of Unapproved Containers (storage areas only) —
The total number of plastic bags, paper sacks, wooden or
cardboard boxes, and so forth, which are used as containers
for solid waste materials.
Number of Non-Complying Cans or Bins (storage areas
only) — The total number of cans or bins found in the stor-
age area that do not comply with established city regulations.
For the field test city, these regulations required that cans
and bins:
— be constructed of metal
— have tight fitting lids (if filled or partially filled)
— be free of holes
— be not larger than 20 gallons.
Number of Cans or Bins in Poor Condition (storage areas
only) — The number of cans or bins of a combustible nature
and/or having holes observed in the storage area. A com-
bustible can is one that is not constructed of metal.
Number of Cans With Size Violations (storage areas only) —
The number of cans larger than the allowable size observed
in the storage area. For the field test city, any can larger
than 20 gallons was considered a size violation.
Number of Cans or Bins Without Tight Lid (storage areas
only) — The number of cans or bins that lack tight lids.
Included would be cans or bins with no lids, cans or bins
with bent or obviously loose lids, and overpacked cans or
bins for which the lid will not fit tightly.
Odors (storage areas only) — Used to indicate the presence
of odors due to poor storage conditions.
141
-------
VACANT LOTS, PUBLIC PARKS, AND PARKING LOTS
FORM
The Vacant Lots, Public Parks, and Parking Lots Data Form
is intended for use in the areas defined as follows:
Vacant Lots: Open spaces that serve no business or resi-
dential purpose.
Public Parks: Playgrounds, recreational, or scenic areas
serving the public.
Parking Lots: Paved open areas, usually having lined spaces
and identifying signs, that are used as places for parking
vehicles.
A replica of this form is provided in Figure D-4 on the following
page.
All vacant lots, public parks, and parking lots within the
boundary of the designated block were to be inspected with the
use of this form. Data for each vacant lot, public park, or park-
ing lot inspected was to be recorded on a separate line on the form.
The format of this form is also similar to that of the two
previous forms. A summary description of the entries that are
unique to this form is provided below.
Sample Area Number — A pre-recorded number that was
used as a reference number for each separate vacant lot,
public park, or parking lot being inspected.
Blockface Number — The identifying number of the block-
face upon which the lot or park is located.
Lot Description Code — A code number used to identify the
inspection area; it was recorded as follows:
1 = Vacant Lot
2 = Public Park
3 = Parking Lot.
142
-------
Form [3] PageO of | |
i 2 3
^_^
Inspector I.D. No. I I I
15 16
Tract No. I I I I 1*1 I I Block
4 5 6 7 8 9 1O 11 12 13
VACANT LOTS, PUBLIC PARKS, AND PARKING LOTS
Mo. Day Vaar
Team No. D Date CD |~T1 |"T~I Day G
17 18 19 20 21 22 23 24
Sanitation District No. I I
14
0 1
1 5
25 26 27 28 29
'Descriptions (reference to Sample Area Number and to Column Number):
Figure D~4: Replica of the Vacant Lots,
Public Parks, and Parking Lots Data Form.
-------
THE SUMMARY FORM
The Summary Form is used to provide information on col-
lection and cleaning activities as they relate to the day that the
inspection was made and on block conditions relative to collection
and cleaning activities. It also provides an estimate of the length
of time that it took to collect the requested information about the
block. Observers were to complete one copy of this form for each
block they inspected. A replica of this data form is provided in
Figure D-5 on the following page.
The Summary Form is organized as follows:
• Identifying Information — Contained in the upper left-
hand corner of the form.
• Time Since Collection and Cleaning — Contained in the
upper right-hand corner of the form.
• Start and Finish Time for Data Collection — Contained
in the upper middle section of the form.
• Composite Block Rating — Contained in the upper
middle section of the form.
• Selected Observations on Block Conditions — Questions
1 through 7 on the form.
A summary description of information requested on this
rm is provided below:
Identifying Information — Used to identify the observer
making the inspection and the sample block being surveyed.
Time Since Collection and Cleaning — Used to indicate the
elapsed time since regular collection, bulk pick-up, and
street and alley cleaning. This information was recorded
by comparing the day that the inspection was made with the
last scheduled day for collection and cleaning, respectively,
as shown on the Pre-Survey Form. Zero (0) was to be re-
corded if the observer was there on a collection/cleaning
day subsequent to the time the collection or cleaning took
place. If collection or cleaning was ongoing at the time the
observer arrived, he (or she) was to wait and inspect the
area subsequent to the completion of the activity.
144
-------
SUMMARY FORM
Name
Tract No.
Block No.
District No.
No. of Days Since Regular Collection
No. of Days Since Bulk Collection
No. of Days Since Street Cleaning
No. of Days Since Alley Cleaning
Start Time:
Finish Time:
Composite Block Rating
am/pm
am/pm
(1-8)
1. Was this the regular collection day for mixed
refuse?
Yes No
2.
3.
If yes to Q. 1, were you there
before collection ?
after collection?
If yes to Q. 1. identify the place of collection by
blockface (e.g., along alley, at end of alley, at
curbside, from backyard):
1.
2.
3.
4.
5.
6.
Blockfaces
7.
8.
9.
10.
11.
12.
4.
Was there any uneollected refuse along any
blockface or alley?
Yes No
6.
7.
If yes to Q. 4. identify the type of uneollected
refuse (e.g., mixed refuse, bulk items, and
so forth) and the blockface/alley number.
Blockface/Alley No.
Type of Refuse
Were any of the streets or alleys being cleaned
while you were there ?
Yes No
If yes to Q. 6, provide the appropriate blockface
and/or alley numbers on the line below.
Figure D-5: Replica of the Summary Form
-------
Start and Finish Time for Data Collection — Used to indicate
the length of time that it took to collect data for the survey
block.
Composite Block Rating — Recorded on a scale of 1 through
8, with 1 indicating the most favorable overall conditions
and 8 indicating the worst overall conditions. This reflects
the observer's subjective assessment of overall block con-
ditions.
Selected Observations on Block Conditions — Used to obtain
further information about block conditions relative to collec-
tion and cleaning activities.
146
-------
APPENDIX E
DESCRIPTION OF THE ANALYSIS PLAN
This appendix provides a detailed description of how some
of the major findings presented in Chapter VI of the report were
developed. Specifically, it describes the techniques used to:
• Analyze the consistency (or reliability) with which the
measurements were made, particularly the rating
scale measurements because of the higher degree of
subjectivity associated with these types of measure-
ments.
• Estimate the variance components for a census tract
mean.
• Assess the accuracy of the measurements.
• Determine correlations among the six observational
areas and among the variables.
• Develop composite measures of effectiveness.
EVALUATING THE CONSISTENCY AMONG RATERS
Reliability or consistency refers to the ability of two or
more observers to independently assign the same value when
measuring the same phenomena. In assessing the reliability of
the data collected during the field test, separate techniques were
developed for each of the three generic types of measures — rating
scales, counts, and yes-no type measures. These are presented
in the following paragraphs.
Consistency Among Raters — Rating Scales
As described in Chapter V, the observers worked in pairs,
with each person separately recording his (or her) measurements
on the data collection forms. To evaluate the reliability of the
147
-------
rating scale measurements, the mean discrepancy between all
observer pairs at each of the seven scale points was analyzed.
The analysis was performed separately for each type of rating
scale—garbage, glass, and refuse—in each of the six observa-
tional areas where data were collected (blockfaces, alleys, storage
areas, etc.). The results were subsequently combined across the
six observational areas by type of rating to develop the findings
presented in Chapter VI.
In performing the analysis, we hypothesized that a relation-
ship of the following type would exist between the size of the mean
rating difference and the rating scale points.
2.0
V 12 •. C
£ Pi 1'5
W 1'°
S 0.5
1 23456
RATING SCALE POINTS
That is, we expected to find fairly close agreement between ob-
servers for conditions 1 and 7 (the scale extremes); i.e., mean
rating differences approximately equal to zero. Toward the mid-
point of the scale, however, we expected to find much less agree-
ment; i. e., mean differences much greater than zero. We felt
that observers were likely to have no trouble recognizing extremely
good and extremely poor conditions, whereas conditions in between
the two extremes were likely to pose the most problems in terms
of rater agreement.
To test the hypothesis, we let:
value assigned by Rater Q to the i* observation point
(where Rater Q was one member of a rater pair)
148
-------
value assigned by Rater P to the i observation point
(where Rater P was the other member of the rater
pair)
value of the absolute difference between a rater pair
at the ith observation point; i. e., /qi - p^
For each observational area, we first looked at the discrep-
ancies between rater pairs (d^) and assumed that Rater Q was
correct in his assignment of rating values, q^. By cross-tabulating
the discrepancies between rater pairs with the values that Rater Q
assigned, we obtained a matrix that contained:
• Frequencies for each cell
• Frequencies and percents for each row and column
• Values for the mean difference between rater pairs
at each of the 7 rating values, along with the standard
deviations associated with each mean difference.
The matrix looked as follows:
Value
Assigned
by
Rater Q
yFreq,
of Total
Absolute Value of the
Discrepancy Between
Rater Q and Rater P (d
Total
Fre-
_, . ^quency
Percent ^ J
of Total
Mean
Value
Standard
Deviation
149
-------
Our particular interest from an analytical point of view was the
information contained in the last column; namely, the mean dif-
ference by scale point and its standard deviation.
Now, since we had no a priori reason to assume that Rater Q
was the correct rater- we proceeded to look at the discrepancies
between rater pairs, on the assumption that Rater P was correct in
his assignment of rating values, p^. When the discrepancies in the
observations of the two raters were cross-tabulated with the values
assigned by Rater P, we obtained a matrix identical in form to the
one presented above, having Rater P's assigned values.
The mean differences by scale point that resulted from the
two cross-tabulations were combined and a weighted average was
developed. This average reflected the mean difference across all
observer pairs for each scale point of a given rating scale. This
procedure was repeated for each of the six observational areas.
The resultant data were pooled to develop new weighted averages
for the mean rating differences across all observational areas.
The standard error of the mean and the 95 percent confidence interval
about the mean rating differences were then computed for each
scale point.
When a plot of the mean rating differences versus the scale
points was made, it was found to exhibit a curvilinear relationship
somewhat different than originally hypothesized. The plotted data
exhibited the following shape for each of the three rating scales.
23456
RATING SCALE POINTS
150
-------
The data thus revealed fairly close agreement between raters at
the lowest scale point. The amount of variation between raters
tended to increase rapidly between scale points 1 and 3 and then
tended to stabilize at the higher scale points. Thus, the close
agreement that was expected at the upper end of the scale did not
materialize.
The relationship between the mean rating differences and
the rating scale points was then estimated statistically using
least-squares techniques to fit a curve of the form:
Y = a - b/X
where X = the rating scale point
Y - the mean difference associated with the given
scale point.
The resultant regression equations provided a good fit to the data.
All had regression coefficients that were significant at the .001 level.
The procedures described above may be summarized in
mathematical notation for each observational area as follows:
(1) Let: d = absolute difference between Raters 1
and 2 of each rater pair at the i* ob-
servation point, assuming Rater 1 is
correct in his assignment of rating j
d-9. = absolute difference between Raters 1
and 2 of each rater pair at the i^1 ob-
servation point, assuming Rater 2 is
correct in his assignment of rating j
(2) Then, the mean difference for the j rating point on
the scale, assuming Rater 1 of each rater pair is
correct, (m .) is:
m
151
-------
and, the mean discrepancy for the j rating point on
the"scale, assuming Rater 2 of each rater pair is
correct, (m~.) is:
m2j =
(3) The mean discrepancy across all pairs of raters for
the jth rating point (MD ) is:
.
tj n + n2.
(4) And the variance of the mean discrepancy at the j
2 2
n. . SD . + n_. SD_.
, . + n_. - 2
1] 2]
(5) And the 95 percent confidence interval for the mean
discrepancy at the j point is:
MD,. + 1.96
Vn. . + n
1] 2]
Consistency Among Raters — Counts
To assess the reliability of measurements that required the
observers to count the number of similar type items that were
present (e.g., bulk items, abandoned vehicles), the following pro-
cedure was developed: we would determine for each count value
the percent of instances in which there was complete agreement
152
-------
between pairs of raters, the percent of instances in which they
disagreed by 2 counts, and so forth.
The technique proposed to handle this issue was similar to
that proposed to handle the question of consistency when rating
scales were used. The only difference was that instead of looking
at the mean difference for each count value, we would be looking
at percent of observations that differ by 0, 1, 2, etc. at each
count value. Percentages were to be used rather than mean dif-
ferences because, whereas it makes sense to speak of a garbage
rating of 3.5, it makes no sense to speak of 3.5 bulk items.
However, when the data on measurements that required
counts were reviewed, it became apparent that the agreement was
so close that further analysis of the reliability of these measure-
ments was unnecessary. That is, raters were found to be in agree-
ment 97 percent of the time or better when it came to assessing
the number of similar type items that were present.
Consistency Among Raters—Yes-No Type Measurements
To determine the reliability of measurements that required
an assessment of the presence or absence of a given condition
(e.g., rat indicators, odors), chi-square tests were performed.
In all cases the results indicated a close degree association be-
tween the responses of paired observers at the .001 significance
level. The percent of agreement was also quite high—96 percent
of the time or better raters were in complete agreement.
ESTIMATING VARIANCE COMPONENTS FOR A CENSUS TRACT
In making statistical inferences about a geographical area
(e.g., the mean garbage rating for a census block, census tract,
sanitation district, etc. ), it must be recognized that there are a
number of sources of variability that can affect the estimate, rater
inconsistency being only one of these. These other sources of
variability include: variation among points in a block, variation
among blocks in a tract, variation among tracts within a stratum,
etc.
153
-------
To the extent that these other sources of variability are
large relative to the variation caused by rater inconsistency, the
disagreement among raters takes on less significance. For this
reason, the analysis phase of the project included an attempt to
estimate the relative contribution of the following variance com-
ponents to the total variance of a tract mean:
• Variation among blocks in the tract
• Variation among observers
• Random sources of variation.
The data collected for the common tract, where all observers
went each day, were used to perform the analysis.
The technique employed to estimate the above variance com-
ponents may be summarized by the statements provided below.
(1) A model for the observation taken by the j observer
in the i block was postulated:
y.. = u + b. + r. + e..
!J i J ij
where y.. = average block value recorded by
13 the jth rater for the ith block
u = true mean score for the tract
b. = block effect
r. = rater effect
J
e.. = random effect
(2) The distribution was assumed to be such that the ex
pected values and the variances for b^, r.:, and e^
would be as follows:
2
0, a for the b.
b i
154
-------
0, o for the r.
r J
0, cr for the e..
e i]
(3) Thus, an estimate of the mean score for the tract
would be provided by:
y-
rb
where
r
b
average tract value
average block value recorded by
the j""1 observer for the i/"1 block
number of raters
number of blocks
(4) And the variance of y... would be:
a2 = a2
y... b
b
a2 + a2
r e
r rb
(5) By performing an analysis of variance, the expected
value of the mean squares (EMS) associated with the
sources of variation indicated in (4) was obtained.
The relationship between the EMS and the above vari-
ance components is:
For blocks:
For raters:
EMS,
EMS
A2 A2
a + r <7,
e b
£2 , u A2
o + b o
e r
— For the error term: EMS
155
-------
(6) The equations listed in (5) were solved in a simul-
taneous fashion and estimates of a^, OT, and ae
were developed.
(7) The values of the variances obtained in (6) were then
substituted into (4) to estimate the total variance of
y... and its components: a, , ar, and ae .
b r rb
ASSESSING THE ACCURACY OF OBSERVERS
Accuracy refers to the degree to which the measured value
approaches the true value. In the case of the rating scales, the
true value of conditions at a given site would be the average value
across an infinite population of observers looking at the same site.
As an approximation of the true value, the mean rating across the
10 observers can be used.
Thus, in order to get an idea of how accurate each observer
was, we used the data from the common tract where an average
value across 10 observers could be made. Specifically, we com-
pared the tract rating for a given observer with the overall tract
rating that resulted when an average value was estimated across
all observers. The 95 percent confidence interval about each
observer's mean value was also computed. In addition, we com-
pared the average block ratings for each observer with the overall
block ratings.
DETERMINING THE CORRELATIONS
Correlation analysis was used to test the degree of association
between:
Similar variables measured in each of the six obser-
vational areas for which data were collected.
Different variables measured within the same obser-
vational area.
156
-------
This was done in order to determine whether it was possible to
reduce the number of observational areas for which measure-
ments should be made and/or the number of different variables.
In both instances, the correlation matrices were developed
on the basis of the Pearson product-moment correlation coeffi-
cient. This is a measure of association that can be used when
the variables represent at least an ordered continuum of some
kind (low to high, agree to disagree, etc. ). It was selected be-
cause it is more powerful than other tests of association; that
is, the probability of making a correct inference based on the
value of the correlation coefficient is higher for this type of test
than for others.
Correlations Among Variables Across Observational Areas
In measuring the correlations among the six observational
areas—blockfaces, private ways, alleys, storage areas, back-
yards, and lots — the average garbage, glass, and refuse ratings
were computed for each of the 100 blocks that were surveyed.
From these averages, simple correlation coefficients were de-
veloped to reflect the strength of the relationship between the
garbage ratings in blockfaces and the garbage ratings in each of
the other areas. Simple correlation coefficients were also de-
veloped to assess the strength of the relationship between glass
and refuse ratings in blockfaces and those in other observational
areas. In a similar manner, correlations were developed between
ratings in alleys and those in the other observational areas, and
so forth.
Correlations Among Variables in the Same Observational Area
Simple correlation coefficients among the variables were
developed for blockface and alley conditions. The analysis was
performed only for these two observational areas because when
the results of the correlation analysis across observational areas
were reviewed, conditions in these two areas were found to be
related at statistically significant levels to conditions in the other
observational areas included in the field test.
157
-------
DEVELOPING THE COMPOSITE MEASURES
As described in Chapter IV, it was decided to utilize the
following techniques to develop indices of solid waste effective-
ness:
(1) Have the observers assign a composite (or global)
rating to each area they surveyed at the same time
they made measurements of the individual variables.
In assigning the composite rating, the observers were
to use an 8-point scale, with 1 representing the most
favorable overall conditions and 8 representing the
most unfavorable overall conditions.
(2) Use regression techniques to estimate the functional
forms corresponding to the three types of decision
models. In the regressions, the composite rating
was to be used as the dependent variable; the other
variables were to be used as the independent variables.
(3) Compare the resultant regressions to determine the
most appropriate decision model and use this to
formulate an index of solid waste effectiveness.
In implementing this approach, the following general forms
were used to specify the three decision models.
Type of
Model Functional Form
n
Linear Y, = a_ + A-< b „ Z
1 1 i=l il i
n
Conjunctive InY = a + . b.0ln(Z.)
<^ ^ i— 1 i& i
n
Disjunctive LnY = a0 + A* b._ln(^>. - Z.)
o 3 i=l i3 i i
158
-------
where Y = dependent variable
Z^ = independent variables
a. - regression constant
b- = regression coefficients
^. = asymtotic values associated with each inde-
pendent variable for the disjunctive model.
In developing the estimating equations associated with the
above functional forms, we let:
Y = 9 - E (where E was the value of the composite
rating, along a scale of 1-8)
Z. = 8 - X. (where Xj was the value of the measured
variables, each scaled so that the lowest
value was 1 and the highest value was 7)
4>i = 7.07
That is, we used a reverse scaling of the variables.
The estimating equations for the three types of decision
models that resulted from application of least-squares techniques
thus became:
Type of
Model Estimating Equation
n
Linear ^1 = al + i?!^!^"^
n
Conjunctive InY = a + .4* b ln(8-X.)
Z Z 1~ J. l
-------
A
In all of these formulations, the higher the value of Yj, the
better the overall conditions. In the case of the conjunctive model,
one can say that all of the measured variables (Xj) must be no
worse than a given level. In the case of the disjunctive model,
one can say at least one of the measured variables must be no
worse than a given level.
Stepwise linear regression techniques were used in estima-
ting the equations. Variables significant at the .05 level or greater
were retained. The equations were estimated both for selected
observers and for the group as a whole. The latter estimations
were developed by averaging the values that individual observers
assigned to the areas they inspected in the common tract.
The resultant equations were then transformed into indices
that could take on values between 0 and 1. This was done by first
finding the maximum value for YJ; i. e., the value that would be
obtained when the X^ values are at their minimum. This was
assigned the value of Yj(max). Next, the value for Yj(mm\ was
found; i. e., the value that would be obtained when the Xj values
are at their maximum. We then set Yj(min) = zero and Yj(max) = 1.
The intermediate scale values were determined by application of
the following formula:
A
\ /
j j(min))//
'/ ( "j(max) J"j(min)
Y - Y 1
-*• -i \ i ./ . \ I
A
where Y. = value predicted by the regression equation
3 (j = 1, 2, 3).
160
-------
APPENDIX F
TABLES, CHARTS, AND GRAPHS
THAT SUPPORT FINDINGS
This appendix contains a number of detailed charts, tables,
and graphs that support the findings presented in Chapter VI. The
tabular displays are presented first, followed by figures that con-
tain the charts and graphs.
161
-------
Table F-l. FREQUENCY DISTRIBUTION FOR BULK ITEMS
BY AREA OF OBSERVATION—ALL TRACTS IN SAMPLE
Area of
Observation
Blockfaces
Private Ways
Alleys
Storage Areas
Backyards
Lots
%
Where
None
Observed
99.6
95.7
83.0
92. 3
80. 6
69.0
%
Where
1
Observed
0.3
2.3
9.3
3.2
8.4
6.9
%
Where
2-5
Observed
0.1
1.6
6.1
4.0
7.2
13.8
%
Where More
Than 5
Observed
0.0
0.4
1.6
0.5
3.8
10.3
Total
Observations
747
735
313
572
692
58
Table F-2. FREQUENCY DISTRIBUTION FOR BULK ITEMS
BY AREA OF OBSERVATION —COMMON TRACT ONLY
Area of
Observation
Blockfaces
Private Ways
Alleys
%
Where
None
Observed
99.4
93.6
72.5
%
Where
1
Observed
0.6
4.7
12.1
%
Where
2-5
Observed
0.0
1.7
15.4
%
Where More
Than 5
Observed
0.0
0.0
0.0
Total
Observations
348
343
182
162
-------
Table F-3. FREQUENCY DISTRIBUTION FOR DEAD ANIMALS
BY AREA OF OBSERVATION—ALL TRACTS IN SAMPLE
Area of
Observation
Bio ckf aces
Private Ways
Alleys
Storage Areas
Backyards
Lots
%
Where
None
Observed
99.3
99. 3
97.8
99.8
99.7
98. 3
%
Where
1
Observed
0.7
0.7
1.6
0.2
0.3
1.7
%
Where
2-5
Observed
0.0
0.0
0.6
0.0
0.0
0.0
%
Where More
Than 5
Observed
0.0
0.0
0.0
0.0
0.0
0.0
Total
Observations
747
735
313
574
694
58
Table F-4. FREQUENCY DISTRIBUTION FOR DEAD ANIMALS
BY AREA OF OBSERVATION—COMMON TRACT ONLY
Area of
Observation
Blockfaces
Private Ways
Alleys
%
Where
None
Observed
99.1
98. 5
92.9
%
Where
1
Observed
0.9
1.5
5.5
%
Where
2-5
Observed
0.0
0.0
1.6
%
Where More
Than 5
Observed
0.0
0.0
0.0
Total
Observations
348
343
183
163
-------
Table F-5. FREQUENCY DISTRIBUTION FOR ABANDONED VEHICLES
BY AREA OF OBSERVATION—ALL TRACTS IN SAMPLE
Area of
Observation
Bio ckf aces
Private Ways
Alleys
Storage Areas
Backyards
Lots
%
Where
None
Observed
100.0
100.0
97.7
NA
98.3
86.2
%
Where
1
Observed
0.0
0.0
1.0
NA
1.7
10.3
%
Where
2-5
Observed
0.0
0.0
1.3
NA
0.0
3.5
%
Where More
Than 5
Observed
0.0
0.0
0.0
NA
0.0
0.0
Total
Observations
747
735
313
NA
694
58
Table F-6. FREQUENCY DISTRIBUTION FOR ABANDONED VEHICLES
BY AREA OF OBSERVATION—COMMON TRACT ONLY
Area of
Observation
Bio ckf aces
Private Ways
Alleys
%
Where
None
Observed
100.0
100.0
98.9
%
Where
1
Observed
0.0
0.0
1.1
%
Where
2-5
Observed
0.0
0.0
0.0
%
Where More
Than 5
Observed
0.0
0.0
0.0
Total
Observations
348
343
183
164
-------
Table F-7. FREQUENCY DISTRIBUTION FOR CLOGGED DRAIN BASINS
BY AREA OF OBSERVATION—ALL TRACTS IN SAMPLE
Area of
Observation
Blockfaces
Alleys
Where
None
Observed
93.3
98.7
Where
1
Observed
6.6
1.3
Where
2-5
Observed
0.1
0.0
Where More
Than 5
Observed
0.0
0.0
Total
Observations
746
313
Table F-8. FREQUENCY DISTRIBUTION FOR CLOGGED DRAIN BASINS
BY AREA OF OBSERVATION—COMMON TRACT ONLY
Area of
Observation
Blockfaces
Alleys
Where
None
Observed
90.2
100.0
Where
1
Observed
9.8
0.0
Where
2-5
Observed
0.0
0.0
Where More
Than 5
Observed
0.0
0.0
Total
Observations
348
183
165
-------
Table F-9. FREQUENCY DISTRIBUTION FOR FIRE HAZARDS
BY AREA OF OBSERVATION—ALL TRACTS IN SAMPLE
Area of
Observation
Blockfaces
Private Ways
Alleys
Storage Areas
Backyards
Lots
%
Where No
Fire Hazard
Observed
100.0
99.6
94.6
98.3
94.4
93.1
%
Where Minor
Fire Hazard
Observed
0.0
0.3
4.5
1.0
4.0
6.9
%
Where Major
Fire Hazard
Observed
0.0
0.1
1.0
0.7
1.6
0.0
Total
Observations
747
735
313
574
694
58
Table F-10. FREQUENCY DISTRIBUTION FOR FIRE HAZARDS
BY AREA OF OBSERVATION —COMMON TRACT ONLY
Area of
Observation
Blockfaces
Private Ways
Alleys
%
Where No
Fire Hazard
Observed
100.0
100.0
88.6
%
Where Minor
Fire Hazard
Observed
0.0
0.0
10. 9
%
Where Major
Fire Hazard
Observed
0.0
0.0
0.5
Total
Observations
348
343
183
166
-------
Table F-ll. FREQUENCY DISTRIBUTION FOR RAT INDICATORS
BY AREA OF OBSERVATION —ALL TRACTS IN SAMPLE
Area of
Observation
Blockfaces
Private Ways
Alleys
Storage Areas
Backyards
Lots
%
Where No Rat
Indicators
Present
98.9
93.2
67.4
91.3
81.4
72.4
%
Where Rat
Indicators
Present
1.1
6.8
32. 6
8.7
18.6
27. 6
Total
Observations
747
735
313
574
694
58
Table F-12. FREQUENCY DISTRIBUTION FOR RAT INDICATORS
BY AREA OF OBSERVATION—COMMON TRACT ONLY
Area of
Observation
Blockface
Private Ways
Alleys
%
Where No Rat
Indicators
Present
97.7
80. 5
29.5
%
Where Rat
Indicators
Present
2.3
19.5
70. 5
Total
Observations
348
343
183
167
-------
Table F-13. FREQUENCY DISTRIBUTION FOR INSECT INDICATORS
BY AREA OF OBSERVATION—ALL TRACTS IN SAMPLE
Area of
Observation
Storage Areas
Where No Insect
Indicators
Present
99.8
Where Insect
Indicators
Present
0.2
Total
Ob s ervations
574
Table F-14. FREQUENCY DISTRIBUTION FOR ODORS
BY AREA OF OBSERVATION—ALL TRACTS IN SAMPLE
Area of
Observation
Storage Areas
%
Where No Odors
Present
96.0
%
Where Odors
Present
4.0
Total
Observations
574
168
-------
Table F-15. GARBAGE RATING SUMMARY STATISTICS
FOR CONSISTENCY AMONG RATERS
Scale
Point
1
2
3
4
5
6
7
Mean
Rating
Difference
.1360
.7022
1.1020
1.4000
1.5780
1.1819
1.5909
Variance
a2
.2676
.5025
1.0298
. 8000
1.8403
1.1637
2. 9840
Standard
Deviation
a
.5173
.7089
1.0148
.8944
1.3566
1.0788
1.7274
95% Confidence
Interval About
the Mean
Difference
+.0143
+ .0766
+.0813
+.3333
+.2142
+.7151
+. 4254
Total
Number of
Comparisons
5052
329
598
30
154
11
66
169
-------
Table F-16. GLASS RATING SUMMARY STATISTICS
FOR CONSISTENCY AMONG RATERS
Scale
Point
1
2
3
4
5
6
7
Mean
Rating
Difference
. 2599
.7871
1.0956
1.1899
1.1975
1.0308
1.5108
Variance
a2
. 5439
. 6702
1.4112
.7791
1.3714
. 9679
3.1741
Standard
Deviation
a
.7375
.8187
1.1879
.8827
1.1711
.9838
1.7816
95% Confidence
Interval About
the Mean
Difference
+.0247
+.0733
+.0664
+.1376
+.1047
+. 2442
+.1705
Total
Number of
Comparisons
3410
749
1224
158
481
65
419
170
-------
Table F-17. REFUSE RATING SUMMARY STATISTICS
FOR CONSISTENCY AMONG RATERS
Scale
Point
1
2
3
4
5
6
7
Mean
Rating
Difference
.4163
. 8705
. 8694
1.1173
1. 3270
1.1409
1. 2463
Variance
a2
.7957
. 9282
1.1119
.8319
1. 2424
1.1228
2. 5734
Standard
Deviation
a
.8920
.9634
1.0545
. 9121
1.1146
1.0596
1 . 6042
95% Confidence
Interval About
the Mean
Difference
+.0376
+ .0710
+. 0504
+.1082
+. 0808
+. 2'±89
+.1278
Total
Number of
Comparisons
2162
710
1684
0 7 o
£j t jj
731
71
605
171
-------
Table F-18. BLOCKFACE ESTIMATING EQUATIONS
FOR LINEAR, CONJUNCTIVE, AND DISJUNCTIVE
MODELS FOR OBSERVER 1
Type of
Model
Linear
Conjunctive
Disjunctive
A
E =
A
lnE =
A
lnE =
a
Estimating Equations
(all coefficients significant at . 001
-.48 + .5lX1 + -45X +
1 "
-.14 +.56 In (X, ) + .441n(XJ +
1 &
2.93 - .19 In (7.07-X, ) - .24 In (7.07-XJ -
JL &
level)
.25X3
,191n(X3)
.841n(7.07-X )
•o
H2
.911
.845
.594
a A
E = blockface composite rating; Xj = blockface refuse rating;
glass rating; Xs = blockface garbage rating.
= blockface
Table F-19. BLOCKFACE ESTIMATING EQUATIONS
FOR LINEAR, CONJUNCTIVE, AND DISJUNCTIVE
MODELS FOR OBSERVER 2
Type of
Model
Linear
Conjunctive
Disjunctive
(all
A
E =
A
lnE =
A
lnE =
a
Estimating Equations
coefficients significant at . 001 level)
.72 + .45X1 +
.27 + .38 In (X1) +
2.17 - .58 In (7.07-Xj) -
.25X2
.34 In (X2)
.30 In (7.07-X )
Zj
R2
.733
.706
.631
a A
E = blockface composite rating; Xj = blockface refuse rating; X2
blockface glass rating.
172
-------
Table F-20. BLOCKFACE ESTIMATING EQUATIONS
FOR LINEAR, CONJUNCTIVE, AND DISJUNCTIVE
MODELS FOR OBSERVER 6
Type of
Model
Linear
Conjunctive
Disjunctive
(all
A
E
A
InE
A
InE
cL
Estimating Equations
coefficients significant at . 01 level)
= 1.57 + .28X + .26X
J. Zi
= .59 + .23 In (X ) + .291n (X )
= 1.31 - .191n (7.07-X )
<£
R2
.508
.541
.214
E = blockface composite rating; Xi = blockface refuse
rating; X2 = blockface glass rating.
Table F-21. BLOCKFACE ESTIMATING EQUATIONS
FOR LINEAR, CONJUNCTIVE, AND DISJUNCTIVE
MODELS FOR OBSERVER 7
Type of
Model
Linear
Conjunctive
Disjunctive
A
E
A
InE
A
InE
(all
= 1.67 +
= .79 +
= 3.85
a
Estimating Equations
coefficients significant at - 01 level)
.19X1 +.
.231n(X ) + .
O c -V" 4. co v
43 A T . O^A.
& o
251n(X )
lOln (7.07-X ) -1.391n(7.07-Xg)
R2
.595
.529
.449
E = blockface composite rating; Xj = blockface refuse rating;
X2 = blockface glass rating; X3 = blockface garbage rating.
173
-------
Table F-22. BLOCKFACE ESTIMATING EQUATIONS
FOR LINEAR, CONJUNCTIVE, AND DISJUNCTIVE
MODELS FOR OBSERVER 10
Type of
Model
Linear
Conjunctive
Disjunctive
(all
A
E =
InE =
A
InE =
a
Estimating Equations
coefficients significant at . 001 level)
.31 + .39 X + .22 X2
-.02 + .42 In (X) + .281n(X)
J. ^
2.13 - .321n (7.07-X ) - .771n (7.07-X )
-L &
R2
.692
.654
.539
a A
E = blockface composite rating; X]_ = blockface refuse rating;
X2 = blockface glass rating.
17'
-------
tf
W
CO
n
o
s
H
fc
o
H
U
P3
100
90
CO
§ "
70
60
50
40
30
20
10
Garbage
Rating
80.9
Glass
Rating
54.7
Refuse
Rating
38.4
1 2,3 4,5 6,7 1 2,3 4,5 6,7 1 2,3 4,5 6,7
RATING SCALE POINTS
Figure F-l: Frequency Distribution for Garbage, Glass, and
Refuse Rating Scales — All Tracts in Sample
175
-------
CO
I
«
W
CO
H
O
H
fe
O
EH
13
H
O
rt
W
100
90
80
70
60
50
30
20
10
Garbage
Rating
63.0
r
5.7
2.8
Glass
Rating
40.3
15.0
L
Refuse
Rating
41.2
1 2,3 4,5 6,7 1 2,3 4,5 6,7 1 2,3 4,5 6,7
RATING SCALE POINTS
Figure F-2: Frequency Distribution for Garbage, Glass, and
Refuse Rating Scales — Common Tract Only
176
-------
7.00
6.00
w 5.00
o
u.
o
O
CO
- 4.00
O
o
3
cc
<
CD
3.00
2.00
1.00
5
j_
I
T
<
I f
i
T -
Mean Blockface
Garbage Rating
-r* Across All Raters
t I T I i--
456
RATERS
10
Figure F-3: Average Garbage Rating for Blockfaces in the Common
Tract: Overall and by Rater (including 95% confidence
interval about the rater means)
177
-------
7.00
f/J
111
o
o
o
00
o
cc
8
6.00
5.00
4.00
3.00
2.00
1.00
Mean Blockface
Glass Rating
Across All Raters
= 3.42
456
RATERS
10
Figure F-4: Average Glass Rating for Blockfaces in the Common
Tract: Overall and by Rater (including 95% confidence
interval about the rater means)
178
-------
7.00
6.00
_ 5.00
co
o
o
m 4.00
GO
D
u.
LII
DC
3.00
2.00
1.00
Mean Blockfaca
Refuse Rating
Across All Raters
3.51
0123456789 10
RATERS
Figure F-5r Average Refuse Rating for Blockfaces in the Common
Tract: Overall and by Rater (including 95% confidence
interval about the rater means)
179
-------
<
cc
cc
<
CD
<
111
7.00
6.00
5.00
4.00
3.00
2.00
1.00
123456
RATERS
Mean Alley
Garbage Rating
Across All Raters
3.15
10
Figure F-6: Average Garbage Rating for Alleys in the Common
Tract: Overall and by Rater (including 95% confidence
interval about the rater means)
180
-------
7.00
6.00
5.00
CO
4.00
co
CO
a
<
LJJ
3.00
2.00
1.00
Mean Alley
Glass Rating
Across Raters
= 5.02
456
RATERS
10
Figure F-7: Average Glass Rating for Alleys in the Common
Tract: Overall and by Rater (including 95% confidence
interval about the rater means)
181
-------
<
cc
CO
LL
UJ
DC
<
LLI
7.00
6.00
5.00
4.00
3.00
2.00
1.00
Mean Allay
Refuse Rating
Across Raters
= 5.31
01 23456789 10
RATERS
Figure F-8: Average Refuse Rating for Alleys in the Common
Tract: Overall and by Rater (including 95% confi-
dence interval about the rater means)
182
-------
TECHNICAL REPORT DATA
(Please read Instructions on the reverse before completing)
. REPORT NO
EPA-670/2-74-082
3. RECIPIENT'S ACCESSI OWNO.
4. TITLE AND SUBTITLE
MEASURES OF EFFECTIVENESS FOR REFUSE STORAGE,
COLLECTION, AND TRANSPORTATION PRACTICES
5. REPORT DATE
November 1974;Issuing Date
6. PERFORMING ORGANIZATION CODE
7. AUTHOR(S)
Messer Associates, Inc.
8. PERFORMING ORGANIZATION REPORT NO.
9. PERFORMING ORGANIZATION NAME AND ADDRESS
Messer Associates, Inc.
8555 16th Street, Suite 706
Silver Spring, Maryland 20910
10. PROGRAM ELEMENT NO.
1DB063/ROAP 02AAE/Task 06
11. CONTRACT/JSAAflW NO.
68-03-0260
12. SPONSORING AGENCY NAME AND ADDRESS
National Environmental Research Center
Office of Research and Development
U.S. Environmental Protection Agency
Cincinnati, Ohio 45268
13. TYPE OF REPORT AND PERIOD COVERED
Final
14. SPONSORING AGENCY CODE
15. SUPPLEMENTARY NOTES
16-ABSTRACTPerhaps between 75 t"o 80 percent of a solid waste system cost is
due to storage, collection, and transportation, the remainder being at-
:ributable to disposal. Given an adequate accounting system, the mone-
:ary costs of a solid waste management system are much easier to compute
than are the benefits produced and the nonmonetary cost incurred. Thus,
although a community may have an accurate estimate of what it is spend-
ing on its system, it often is uncertain as to whether or not it is
receiving reasonable value in the benefits returned; i.e., it has little
or no idea of its "cost effectiveness." This report presents the result
of a project that focused on the systematic development of a set of meas-
ures and measurement tools that could be used to assess the effectiveness
of solid waste storage, collection, and transportation practices. The
project included a pilot test of the measurement methodology in an urban
community. The measurement system presented in this report is intended
o support municipal decision-makers who have responsibility for such
services as mixed refuse collection, street and alley cleaning, sanitary
code enforcement, sanitation education, and other related activities. It
provides a model or prototype that municipal representatives can use to
design effectiveness measures that are specific to their own solid waste
management needs and activities. The report includes a comprehensive
List of candidate effectiveness measures along with the measurement tech-
liques and sampling procedures needed to collect data to formulate the
Candidate measures.
17.
KEY WORDS AND DOCUMENT ANALYSIS
DESCRIPTORS
b.IDENTIFIERS/OPEN ENDED TERMS
COSATI Field/Group
*Measurement
Refuse
Storage
Collection
Transportation
*Solid waste
ment
*Effeetiveness
urements
manage-
meas -
13B
18. DISTRIBUTION STATEMENT
RELEASE TO PUBLIC
19. SECURITY CLASS (This Report)
UNCLASSIFIED
21. NO. OF PAGES
201
20. SECURITY CLASS (This page)
UNCLASSIFIED
22. PRICE
EPA Form 2220-1 (9-73)
183
U. S. GOVERNMENT PRINTING OFFICE: 197't-657-586/53n Reg ion No. 5-1
------- |