Guide for Measuring Compliance
Assistance Outcomes
Revised October 2007
Office of Enforcement and
Compliance Assurance
Washington, DC 20460
EPA 300-B-07-002
October 2007
www.epa.gov/compliance/assistance
-------
Page 1 Table of Contents
TABLE OF CONTENTS
Page
SECTIONI: INTRODUCTION 1
A. Background 1
B. History and Purpose of This Guide 1
C. Overview of Guide 2
SECTION II: COMPLIANCE ASSISTANCE ACTIVITIES AND OUTCOMES 4
A. Compliance Assistance Activities 4
B. Measuring Results of Compliance Assistance 5
C. Continuum of Output and Outcome Measures 7
SECTION III: HOW TO PLAN AND DESIGN AN ASSESSMENT 8
A. What Is the Goal of Your Compliance Assistance Project? 8
B. What Is the Purpose and Scope of Your Evaluation? 9
C. What Measures Are Appropriate? 12
D. Which Data Collection Method Best Meets Your Needs? 13
E. Do You Need an ICR? 18
SECTION IV: HOW TO GET THE MOST OUT OF YOUR SURVEY 21
A. Questionnaire Design 21
B. Survey Implementation-The Tailored Design Method 29
C. Where Do We Go From Here? 38
D. Lessons Learned from Measurement Projects 46
SECTION V: AN INTRODUCTION TO STATISTICAL SAMPLING 48
A. Key Terms 49
B. Statistical Validity 50
C. Sampling Frame-The List of Potential Respondents 50
D. Estimating A Sample Size 53
E. Drawing the Sample From the Population 60
SECTION VI: OECA' S EXPERIENCE WITH THE DILLMAN METHOD AND
STATISTICALLY VALID SURVEYS 62
A. Background and Purpose of the Study 62
B. Methodology 64
C. Implementation 68
D. Results 69
E. Lessons Learned 86
-------
Table of Contents
Page 2
APPENDIX A:
APPENDIX B:
APPENDIX C:
APPENDIX D:
APPENDIX E:
APPENDIX F:
Performance Profile for EPA's Enforcement and Compliance
Assurance Program
Office of Compliance Guidance on the Need for Information
Collection Requests (ICRs) for the Evaluation of Compliance
Assistance Activities
How to Obtain Clearance for Regional Compliance Assistance
Evaluation Surveys Under the Generic ICR 1860.01 OMB Control
#2020.0015
Menu of Sample Survey Questions by Outcome Measure
Examples of Letters Used by OECA to Gather Data by the Dillman
Method
Advanced Data Analysis - Measuring Association and Making
Comparisons
APPENDIX G:
Sampling From Unknown Populations
-------
Page 3 Table of Contents
LIST OF FIGURES
Page
2-1 Continuum of Compliance Assistance Measures 7
3-1 Do You Need An ICR? 19
4-1 Example Pie Chart for Frequency Distribution 40
4-2 Example Chart of Frequency Distribution 41
4-3 Example of Answer Distribution Chart 43
5-1 Formulas for Estimating Sample Sizes 59
6-1 Metal Finishing Performance Evaluation Survey 90
6-2 Marina Checklist 95
LIST OF TABLES
Page
3-1 Outcome Measures for Compliance Assistance 12
3-2 Pilot Project Response Rates 13
3-3 Compliance Assistance Data Collection Tools 16
3-4 OECA's Recommended Follow-up Methods by Assistance Activity 17
4-1 Subset of Dillman's Principles of Questionnaire Construction 28
4-2 Example Frequency Distribution 40
6-1 Background Information on EPA Data Collection Study 63
6-2 Factors Contributing to Sample Size Estimate 64
6-3 Response Rates 71
-------
Table of Contents Page 4
6-4 Mailed Survey Returns 72
6-5 Distribution of Responses for Site Visits 73
6-6 Cost Per In-Scope Response 74
6-7 General Characteristics of the Two Samples in the Metal Finishing Sector 76
6-8 Types of Compliance Assistance Received For the Two Samples in the Metal Finishing
Sector 76
6-9 Awareness of Regulatory Requirements for the Two Samples in the Metal Finishing
Sector 77
6-10 Characteristics of Marinas in Both Samples 78
6-11 Compliance Comparison for Metal Finishing Sector 80
6-12 Summary of Compliance Question Results for Metal Finishing - Comparison of Mailed
Surveys and On-Site Observation Compliance Rates 81
6-13 Hazardous Waste Compliance for Marinas 82
6-14 Oil and Fuel Compliance for Marinas 83
6-15 Hazardous Materials Compliance for Marinas 84
6-16 Storm Water Compliance for Marinas 84
6-17 Summary of Compliance Question Results for Marinas - Comparison of Mailed Surveys
and On-Site Observation Compliance Rates 85
-------
Section I: Introduction Page 1
SECTION I: INTRODUCTION
A. Background
Congress enacted the Government Performance and Results Act (GPRA) in 1993 to
encourage federal managers to measure success based on the results of their projects instead of
simply the level of activity occurring under their programs. The act requires all federal agencies
to develop 5-year strategic plans with goals, objectives, and performance measures. In response
to GPRA, EPA developed a strategic plan that delineates goals to achieve its objectives. EPA
submitted the Agency's 2006-20011 Strategic Plan to Congress in September 2006. This revised
Strategic Plan contains five goals. Goal 5, Compliance and Environmental Stewardship requires
the Agency to ensure full compliance with laws intended to protect human health and the
environment. EPA plans to achieve this goal through compliance incentives and assistance
programs, identifying and reducing significant noncompliance in high-priority program areas,
and maintaining a strong enforcement presence in all regulatory program areas.
The Office of Enforcement and Compliance Assurance (OECA) has developed a set of
measures to evaluate its performance toward achieving Goal 5. This set of measures includes
outcome measures (changes in behavior due, at least in part, to compliance assurance activities),
environmental indicators (measures of progress toward achieving environmental or human health
objectives), as well as output measures (measures of the numbers of activities). These measures
apply only to EPA's federal enforcement and compliance assurance program. They do not serve
as a framework for measuring performance of state enforcement and compliance assurance
programs. OECA and the states developed a separate set of accountability measures for state
enforcement and compliance assurance programs, incorporated in Performance Partnership
Agreements (PPAs). Thus, while OECA's measures are not applicable to states directly, states
can still find information in this guide to help measure goals articulated in their PPAs. Appendix
A includes a sample performance profile for EPA's Enforcement and Compliance Assurance
program.
-------
Page 2 Section II: Compliance Assistance Activities and Outcomes
B. History and Purpose of This Guide
OECA first published this document in March 1999 to help Regions measure the results
of compliance assistance being delivered through a series of pilot projects. Since that time,
Regions and states have gained a great deal of experience measuring results. In addition, EPA
received feedback on the 1999 guide from the Compliance Assistance Advisory Council
(CAAC), participants in Regional training programs, and other stakeholders, and issued a
revised version of the Guide for Measuring Compliance Assistance Outcomes in June 2002.
The 2002 version responded to that feedback and incorporated lessons learned from the federal
pilot projects and state measurement projects, and added information on how to conduct
statistical studies.
The 2007 revision of the guide updates descriptions of compliance assistance activities,
measures and background information, updates EPA contacts and EPA Web sites on gathering
measurement data, when an Information Collection Request (ICR) is needed, and also updates
cost estimates for statistically valid surveys. Other sections of the guide on planning and
designing an assessment, getting the most out of your survey, and the introduction to statistical
sampling, and OECA's experience with statistically valid surveys, remain unchanged from the
2002 version.
OECA believes that measuring results is a key step to making better management
decisions and complying with GPRA. Evaluations conducted under this guide are consistent with
GPRA requirements1 and are compatible with the requirements of EPA's Integrated Compliance
Information System (ICIS). In addition, EPA has incorporated lessons learned from these pilot
projects into its strategic plan and annual performance plans. EPA management has also used the
results of measurement projects to support internal management and policy decisions.
1 Prior to publishing this revised guide, EPA had not used the results of the pilot projects for GPRA purposes due to
concerns of the subjectivity of previous surveys. Through this guide and related projects, OECA is putting forth a
renewed effort to collect statistically valid data and to improve the quality of data collected through nonstatistical
methods.
-------
Section I: Introduction Page 3
C. Overview of Guide
This guide consists of the following sections designed to help you measure the outcomes
of compliance assistance:
Section II: Compliance Assistance Activities and Outcomes discusses the
types and purposes of compliance assistance activities and the outcomes
associated with these activities.
Section III: How to Plan and Design an Assessment provides a step-by-step
overview of key planning and design issues and describes the benefits and
limitations of different data collection tools.
Section IV: How To Get The Most Out Of Your Survey includes tips on
writing good surveys and suggestions on how to get more out of your evaluation
using the Tailored Design Method.
Section V: An Introduction To Statistical Sampling provides a simplified
discussion of how to conduct a statistically valid evaluation.
Section VI: OECA's Experience with the Dillman Tailored Design Method
discusses the recent Metal Finishing and Marina Compliance Assistance Program
Evaluation Study.2
2 Don Dillman is the Thomas S. Foley Distinguished Professor of Government and Public Policy in the Departments
of Sociology and Community and Rural Sociology, and Deputy Director for Research and Development in the
Social and Economic Sciences Research Center (SESRC) at Washington State University. The Tailored Design
Method is a comprehensive approach to designing and implementing self-administered surveys. A complete
exposition of the method can be found in Don Dillman's book Mail and Internet Surveys: The Tailored Design
Method (John Wiley and Sons Inc., 1999).
-------
Page 4 Section II: Compliance Assistance Activities and Outcomes
SECTION II: COMPLIANCE ASSISTANCE ACTIVITIES AND
OUTCOMES
Compliance assistance includes activities, tools or technical assistance that provides clear and
consistent information for 1) helping the regulated community understand and meet its obligations under
environmental regulations; or 2) compliance assistance providers to aid the regulated community in
complying with environmental regulations. Compliance assistance may also help the regulated
community find cost-effective ways to comply with regulations and/or go "beyond compliance" through
the use of pollution prevention, environmental management practices, and innovative technologies, thus
improving their environmental performance. To be categorized as a compliance assistance project or
activity, at least one objective must be related to achieving or advancing regulatory compliance.
A. Compliance Assistance Activities
For purposes of tracking and reporting compliance assistance activities in ICIS, EPA has
identified the following activity types: facility visits and revisits, ongoing facility specific work,
workshops/training, presentations/meetings, public outreach distributed, targeted outreach distributed, and
responses to inquiries. Included here are the general definitions for each of these activity types.
Additional examples and protocols related to entering data on these activities in ICIS can be found in the
ICIS Policy on Demand Database (IPOD), a Lotus Notes-based application located on the desktop. IPOD
can be used by anybody with access to EPA's Lotus Notes system.
Facility Visit: On-site visits where the primary purpose is to provide environmental
assistance to help regulated entities understand and comply with environmental
requirements. Such on-site facility visits are also termed Compliance Assistance Visits
(CAV). Assistance delivered in the context of a compliance inspection is not a CAV, but
should be documented as a compliance monitoring activity on the Inspection Conclusion
Data Sheet and in the Compliance Monitoring module of ICIS.3
Facility Revisit: A follow-up on-site visit to a facility that was previously visited to
provide compliance assistance. The purpose of the visit is generally to provide additional
compliance assistance or to assess progress in implementing improved environmental
management practices or pollution reduction opportunities identified in the previous visit.
3Memorandum, Fiscal Year 2005 Inspection Conclusion Data Sheet (ICDS) Expansion, Implementation and
Reporting, Michael Stahl, September 10, 2004
-------
Section II: Compliance Assistance Activities and Outcomes Page 5
On-going Facility Specific Work: Working with a regulated entity over time, through a
series of on-site visits or targeted meetings to improve or go beyond compliance with
environmental requirements. Examples may include helping a facility to develop an
environmental management system (EMS), pollution prevention assessments with
follow-up, and implementation of compliance assistance plans with tribes. Sending letters
to the same facility on a regular basis, e.g., Toxic Release Inventory (TRI) reporting
letters, is not considered on-going facility specific work (rather, this fits the definition of
Targeted Outreach).
Tools Developed: These are substantial environmental assistance products that can be
shared across Regions and/or locations. Examples include: creation of newsletters, fact
sheets, information packets, brochures, videos, slide shows, newly developed web sites
(not maintenance of existing web sites), plain language guides, case studies, self-audit
checklists, expert systems, tools on CD- ROM( including those which help facilities
make applicability determinations), or public service announcements. Support documents
for workshops, meetings, presentations or targeted outreach, such as briefing materials,
graphs and charts, are not considered to be "Tools Developed" unless they contain
compliance information and are specifically developed for wide distribution as an
environmental assistance product.
Workshops/Training: Activities include seminars, conferences, training, and forums
where the primary focus is to provide environmental assistance to help regulated entities
or other assistance providers understand how to comply with environmental
requirements. These activities are typically 1A day to multiple day sessions. Exhibit
booths are not workshops or training and would be considered either Targeted or Public
Outreach depending on the show. Workshops/training usually require more of a time and
labor investment than Presentations/Meetings.
Presentations/Meetings: Activities include presentations or meetings where the primary
focus of the remarks or discussion is to provide environmental assistance to help
regulated entities or other assistance providers understand how to comply with
environmental requirements. This category is distinguished from a workshop in that it
typically requires less time and labor investment, and is often a smaller part of a larger
effort (e.g., a 2 hour presentation at an all day conference). Also, note that meetings do
not have to occur in person to be counted in this category. Conference calls and video
conference meetings should also be included.
Targeted Outreach Delivered: This activity includes all tools and materials that were
mailed or distributed to a specific audience, such as regulated entities in a particular
sector, or assistance providers/community members and provide a tailored message that
includes compliance with environmental requirements. Examples include downloads
from and user sessions on sector-specific web sites4, compliance articles in trade
4 A "download" occurs when a file is copied from the website server to the user's computer. A "user session" is a
series of actions that begin when a user lands on his/her first page of the website and ends when s/he either leaves
the site or remains idle for more than 30 minutes. The number of sessions may include multiple visits by the same
-------
Page 6 Section II: Compliance Assistance Activities and Outcomes
magazines, and regulation-specific mailings to affected facilities.
• Public Outreach Delivered: This activity includes all compliance assistance tools and
materials that are delivered to a general audience. Examples include public service
announcements (PSAs), articles in general circulation newspapers (not targeted
newsletters), general outreach at home and garden shows (e.g., fact sheets handed out at a
exhibit booth), and downloads and user sessions from general web sites.
• Response to Inquiry: Response to questions regarding compliance with requirements via
phone calls, e-mails, letters, faxes, walk-ins, or contacts through Web pages, etc.
B. Measuring Results of Compliance Assistance
EPA has identified three types of measures to gauge the success of compliance assistance
activities. These measures include output measures, outcome measures, and environmental and public
health impact indicators. While this document focuses on outcome measures, understanding output
measures and environmental and public health impact indicators helps put the outcome measures into
perspective.
Output Measures
Output measures are defined as "quantitative or qualitative measures of important activities, work
products, or actions taken by EPA or by states under delegated federal programs."5 They assess both the
number of EPA products or activities (e.g., the number of fact sheets developed, the number of on-site
visits conducted) and the number of facilities reached through compliance assistance activities (e.g., the
number of helpline calls answered, the number of people at a workshop, the number of guides
downloaded from the Internet).
Outcome Measures
Outcome measures are "quantitative or qualitative measures of changes in behavior of the public
user.
5 U.S. EPA. 1997. National Performance Measures Strategy (NPMS).
December, p. 4.
-------
Section II: Compliance Assistance Activities and Outcomes
Page?
or regulated entities caused, at least in part, by actions of government."6 Outcome measures include
changes in understanding7, changes in behavior, and site-specific environmental and human health
improvements:
• Changes in understanding reflect an increased knowledge of regulatory or
nonregulatory environmental issues, including reporting and monitoring requirements,
regulatory schedules, and pollution prevention opportunities. An example of changes in
understanding includes measuring the percentage of facilities receiving assistance that
indicate an improved understanding of environmental regulations or the number of
facilities attending a workshop that gained knowledge about pollution prevention or
control technologies. Changes in understanding can most effectively be measured by
testing knowledge before and after the
workshop.
Behavioral changes represent
actual changes that a regulated
entity has undertaken as a result
of compliance assistance.
Examples of behavioral
changes include the number of
facilities that submitted
required permit applications or
notification forms because of a
training program, or the
number of facilities that
adopted recommendations
discussed during an on-site
visit. Behavioral changes can
be voluntary (e.g., voluntary
implementation of pollution
prevention technologies as a
result of publication of
pollution prevention guidance
documents or fact sheets) or
regulatory (e.g., facilities
reporting overlooked chemicals
as a result of the publication of
Toxic Release Inventory
EVALUATING BEHAVIORAL CHANGES
- THE NATIONAL NITRATE
COMPLIANCE INITIATIVE
In 2000, EPA implemented the National
Nitrate Compliance Initiative, which educated
facilities in the metal finishing sector about
Emergency Planning and Community
Right-to-Know Act (EPCRA) §313
requirements and EPA's Audit Policy. Through
a phone survey, Region 3 found that 78 percent
of the respondents (14 facilities) indicated a
behavioral change as a result of the mailing.
Examples of actions taken included:
• Conducted an audit;
• Made a process, operating, or material
change;
• Developed an internal monitoring/reporting
system;
• Researched alternatives and substitutions;
and
• Obtained further technical assistance.
6 NPMS, p. 4.
7 The original guide combined awareness and understanding into one measure. Feedback from the Regions
implementing the pilot projects and CAAC suggested that these two measures be separated as they are distinct
conditions. EPA considers changes in understanding more indicative of results than simple changes in awareness.
EPA now considers awareness a component of "reach", an output measure.
-------
Page 8 Section II: Compliance Assistance Activities and Outcomes
guidance documents). Improvements in compliance are also included under behavioral
change. As a group, such changes are often referred to as "improved environmental
management practices."
• Environmental and human health improvements are measures of environmental and
human health improvements at specific facilities resulting from compliance assistance
activities. Examples of environmental and human health improvements would be the
number of pounds of pollutant emission reductions at a facility that adopted a control
technology explained in a training video, or the number of facilities reducing chemical
exposure to workers as a result of practices presented at a workshop.
Environmental and Public Health Impact Indicators
Environmental and public health impact indicators are defined as quantitative or qualitative
measures of progress over time toward achieving national environmental or human health objectives.
These indicators help EPA measure what impacts its environmental programs are having on national
environmental problems. Environmental indicators might, for example, show a reduced level of nutrients
in a water body over a specified amount of time.
C. Continuum of Output and Outcome Measures
Figure 2-1 shows the continuum of output and outcome measures starting from reaching your
targeted population to having the population reduce pollution as a result of your efforts. Each measure
builds on the previous measure on the continuum. Changes in behavior will not occur until the target
audience understands the regulatory requirements. Similarly, it is difficult to assess the site-specific
environmental and human health improvements without knowing the changes in behavior.
Although this document focuses on measuring the outcomes (i.e., changes in understanding,
changes in behavior, and environmental and human health improvements) associated with compliance
assistance projects, understanding how effectively you have reached the target audience will help you
measure these outcomes. If the hotline, assistance tool, or workshop is reaching only a small portion of
the intended audience, there will be limited corresponding changes in understanding and behavior. For
example, if only a few printers in a targeted community are aware of a compliance assistance workbook
and hotline, only a small number of facilities can possibly make changes as a result of the assistance.
-------
Section II: Compliance Assistance Activities and Outcomes
Page 9
Figure 2-1. Continuum of Compliance Assistance Measures
Continuum of Compliance Assistance Measures
Output Mvssur
Reach
Changes in
Understanding
Environmental
Changes in and Human Health
Behavior Improvements
Level of Effort to Measure
OECA's 2005 Guidance Addendum for Reporting Compliance Assistance in the Integrated
Compliance Information System (ICIS), available within the ICIS database, encourages the measurement
of outcomes from direct forms of compliance assistance where there is one-on-one interaction between an
assistance provider and regulated or non-regulated entities, allowing a thorough discussion of compliance
requirements and issues, e.g., on-site facility visit or revisits, ongoing facility specific work, or
workshop/training, presentation/meetings and responses to inquiries. These activities provide the best
opportunities for outcome measurement. Note, that the Guidance Addendum is being updated and will be
reissued in 2008.
-------
Page 10
Section III: How to Plan and Design an Assessment
SECTION III: HOW TO PLAN AND DESIGN AN ASSESSMENT
This section will assist you in planning and designing evaluations of your compliance assistance
projects. This is the most critical stage in evaluating your success. Without effective planning and design,
the subsequent steps in an evaluation are likely to provide results that are inconclusive and difficult to
understand. Identifying the goals of your compliance assistance project and where on the continuum
(Figure 2-1) you are starting from, and defining the purpose and scope of the evaluation, will help you
determine the best approach for your evaluation. To begin planning your evaluation, answer the following
key questions:
• What is the goal of your compliance assistance project?
• What is the purpose and scope of your evaluation? (Should you collect statistical or
anecdotal data?)
• What measures are appropriate to determine if you have met your goal?
• Which data collection method best meets your needs?
• Does the Paperwork Reduction Act apply?
A. What Is the Goal of Your
Compliance Assistance Project?
The first step in the evaluation process is to
identify the goals of your compliance assistance
project. Is the goal of your project to achieve
environmental results, such as improving the storage
and disposal of hazardous waste by regulated
facilities? Or, is the goal to increase understanding of
reporting requirements? Understanding where your
project falls on the compliance assistance continuum
will help ensure that you select appropriate measures
to evaluate your success. For example, through a
2001 pilot project, Region 7 promoted compliance
with Resource Conservation and Recovery Act
REGION 9'S GOAL: IMPROVED
COMPLIANCE IN OPPRESSING
HAZARDOUS AIR POLLUTANTS
In support of one of EPA's national
priorities, Region 9 set a goal to improve
compliance with the Chrome National
Emission Standards for Hazardous Air
Pollutants (NESHAP) at chrome plating
facilities in the Clark County area of Nevada.
The Region provided on-site compliance
assistance tailored to address problems
identified in earlier visits. During a
subsequent return to these facilities,
inspectors were able to accurately measure
the result of the initiative—100 percent
compliance with all parts of the NESHAP at
every facility.
-------
Section III: How to Plan and Design an Assessment Page 11
(RCRA) requirements, including proper storage and disposal methods, to colleges and universities. Since
the goal of this project was to motivate behavioral change (e.g., proper storage of waste) and achieve
environmental outcomes (e.g., proper disposal of waste), measuring both behavioral change and
environmental improvements was appropriate. In another example, Region 5 conducted a pilot project to
improve Toxic Release Inventory (TRI) reporting. The Region's goal was to make sure that facilities
understood how they should properly report. For this project, measuring changes in understanding TRI
reporting requirements and forms before and after the workshop was appropriate.
Examples of specific project goals include:
• Improve the quality of TRI data by helping the regulated community understand the
EPCRA reporting requirements;
• Raise understanding of RCRA requirements via seminars to help prevent improper
hazardous waste management;
• Reduce the high number of chemical accidents in the ammonia refrigeration industry by
providing a sector-specific compliance assistance manual focused on preventing
accidental spills; and
• Assist colleges and universities in meeting their environmental regulatory obligations
through forums on regulations that apply to educational institutions.
For additional information about planning and selecting goals and measures for your project,
consult OECA's Guide for Addressing Environmental Problems: Using an Integrated Strategic Approach
(EPA 305-R-07-001, March 2007)
http://www.epa.gov/compliance/resources/policies/assistance/strategicguide.pdf). It is a detailed
reference for federal, state and tribal agency staff and managers on using a well considered mix of
assistance, inspections, incentives and enforcement to maximize the environmental benefits from their
work. Chapter 2, "Establishing Goals and Measures" and Chapter 8, "Monitoring and Evaluating the
Strategy" of the Guide may be of particular value as you plan your project.
-------
Page 12
Section III: How to Plan and Design an Assessment
B. What Is the Purpose and Scope of Your Evaluation?
Carefully assessing the evaluation's purpose—why the evaluation is needed, how the information
will be used, and who will use the evaluation results—will help you determine the scope of the
evaluation. A key decision in determining the scope is whether to use a statistically valid study or an
anecdotal assessment. For example, if you want to generalize to the overall population or to compare two
groups (such as those that received compliance assistance and those that did not), then you might want to
consider conducting a statistical study. On the other hand, if you are collecting information such as
lessons learned, innovative techniques used by facilities as a result of compliance assistance, or how the
compliance assistance activity helped the audience, then an anecdotal assessment will probably meet your
evaluation needs. Whichever option you choose, consider using observable data (e.g., conduct on-site
revisits or database checks) to document results.
Statistically Valid Evaluations
If you want to generalize to the overall
population or to compare two groups (such as those
that received compliance assistance and those that
did not), you might want to consider conducting a
statistical study. Statistical evaluations enable you to
generalize your evaluation results to a larger
audience (e.g., all facilities in an industrial sector,
geographic region, or all users of a compliance
Consider using a statistically valid
assessment if you want to:
• Generalize results to an overall
population
• Compare two groups
• Evaluate the overall impact of a
program
assistance tool). This type of study requires additional up-front planning in identifying and selecting study
participants, and may require that you collect data from a larger number of respondents to make reliable
generalizations. Statistically valid surveys may require more resources than anecdotal assessments
(discussed below). See Section V for guidance on how to conduct statistically valid evaluations.
-------
Section III: How to Plan and Design an Assessment Page 13
EVALUATING THE IMPACTS OF COMPLIANCE ASSISTANCE ON METAL
FINISHERS
EPA conducted a statistically valid assessment of metal finishers in Regions 1 and 5 to
identity areas requiring additional compliance assistance (e.g., RCRA, air regulations TRI
reporting) and to evaluate the effectiveness of compliance assistance programs. By surveying
a sample of 100 metal finishers, EPA was able to extrapolate results to the two regions.
Anecdotal Assessments
Anecdotal assessments—evaluations that
describe accomplishments, yet make no broad
generalizations or claims8—are suitable for most
• Follow methods to get a high response
compliance assistance evaluations conducted by
OECA. Anecdotal assessments tell a story about how
compliance assistance has impacted the group of ' Want to receive an indication of whether
you have met your goals
people that responded to your survey. These
Consider using an anecdotal assessment if
you:
Do not expect nonresponse bias
evaluations can provide some quantitative information, such as the number of facilities changing behavior
as a result of receiving a compliance assistance tool, and can also ascertain how and why the facilities
responded to the compliance assistance. However, results from these assessments cannot be scaled up to a
larger group. In general, an anecdotal assessment relies on surveys of parties receiving compliance
assistance. One problem of anecdotal assessments is that, unless there is a very high response rate, they
are subject to nonresponse bias (i.e., the results of those who responded may be significantly different
from those who did not respond). To minimize this potential bias, use the data gathering practices
outlined in Section IV, to maximize your response rate. A well-thought-out and we 11-received anecdotal
assessment can provide useful information for making a decision as to whether this activity is worth
continuing, and can be worthwhile if sufficient resources are not available for a statistical study.
8 Sparrow, Malcolm K. 1997. Regulatory agencies searching for performance measures that count. Journal of Public
Administration Research and Theory.
-------
Page 14 Section III: How to Plan and Design an Assessment
INCREASING UNDERSTANDING OF HAZARDOUS WASTE MANAGEMENT ON
THE U.S./MEXICO BORDER
Region 6, in partnership with the Texas Natural Resource Conservation Commission
(TNRCC), found that warehouses along the U.S./Mexico border were violating RCRA
requirements because of a lack of knowledge of RCRA and hazardous waste management. In
response, Region 6 and TNRCC developed a compliance assistance seminar designed to
improve understanding of RCRA. Region 6 conducted a survey of the seminar to learn how to
improve the compliance assistance and to determine whether it was effective and worth
continuing. Results indicated that 86% of respondents would like more similar seminars made
available, and 50% made changes in environmental practices or took other action to comply
with the regulations as a result of the seminar.
Using Observed Data
Whether you intend to conduct a statistically valid study or an anecdotal assessment, you should
consider ways you can directly observe changes made by recipients of your compliance assistance. Direct
observation—measuring changes through pre-and post-tests, observing changes on-site, or verifying that
reporting requirements have been met through database checks—tends to be more reliable than surveys
that simply ask respondents to indicate whether or not they made changes.
You can use direct observation in a variety of ways. For example, for workshops, you might
consider assessing the participants' understanding of regulations before and after the workshop. Region 6
conducted a workshop designed to teach warehouse employees about RCRA requirements and hazardous
waste management. Since the goal was to increase understanding, the Region administered pre-and post-
tests to determine the increase in knowledge of RCRA requirements (the result was a 29 percent increase
in test scores) and receive feedback about the specific strengths and weaknesses of the material presented.
Another option might be to visit facilities to see if they have, in fact, made changes to comply with the
regulations for which they received assistance. Region 8 used inspections to evaluate the effectiveness of
multimedia compliance assistance delivered to auto service and repair shops. During the visits, Region 8
inspectors directly observed and measured increases in compliance and beyond-compliance actions.
Finally, you might consider tracking compliance data that your audience is required to report, such as TRI
reports, EPCRA §312 reports, and permit applications. For example, if you conduct a workshop on air
permitting requirements, check to see how many of the attendees applied for permits 6 months after the
-------
Section III: How to Plan and Design an Assessment Page 15
workshop. Also, check to see if the quality of the reporting has improved.
C. What Measures Are Appropriate?
Once you establish the purpose and scope of the evaluation, you will need to define what
measures you will use to evaluate the success of the program. Defining success up front, based on the
project goal and purpose of the evaluation, will help you select the appropriate evaluation measures, as
well as interpret the results. Try to identify what results you would expect based on your experience with
similar projects and the goals you established at the outset of the activity. Table 3-1 lists specific
measures developed by OECA for assessing outcomes that occurred as a result of compliance assistance
activities to regulated entities; consider this set of measures in developing your project. OECA developed
these measures based on its analysis of the results of pilot projects. OECA tracks most of these measures
on a national level through the Integrated Compliance Information System (ICIS). Use the descriptions of
different measures in Section II to decide which measures best fit your needs and resources.
Also, you should reference the current year's GPRA measures and the current generic ICR for
compliance assistance. The GPRA measures are what EPA will report annually to Congress and the
public. To learn about the current GPRA measures for compliance assistance, go to the enforcement and
compliance sections of EPA's current Strategic Plan at
www.epa.gov/compliance/data/planning/longterm.html. If you plan to measure behavior change resulting
from compliance assistance, you should reference the generic ICR for compliance assistance. It identifies
questions that have been approved by the Office of Management and Budget for collecting data to support
your measures. See Appendix D. Menu of Sample Survey Questions by Outcome Measure, Sections I,
Parts B and C, for ICR questions indicating behavioral and environmental changes that have been
included in our generic ICR for CA. To view the most current generic ICR, go to:
www.epa.gov/compliance/assistance/measures/index.html.
Table 3-1. Outcome Measures for Compliance Assistance
Outcome Measure
Changes in
Understanding
Some Specific Measures
of Outcomes Resulting from Compliance Assistance
Number/percentage of facilities/respondents who say they better
understand how to comply with environmental regulations
-------
Page 16
Section III: How to Plan and Design an Assessment
Outcome Measure
Behavioral
Changes
(also referred to
as "Improved
Environmental
Management
Practices")
Environmental
and Human
Health
Improvements
Some Specific Measures
of Outcomes Resulting from Compliance Assistance
Number of facilities that contacted someone for further compliance
assistance
Number of facilities that took at least one recommended action to comply
with environmental regulations (e.g., labeling, recordkeeping, reporting,
obtaining a permit)
Number of facilities/respondents that adopted at least one process change
(e.g., pollution prevention changes, best management practices)
Number of facilities improving environmental management systems or
conducting reviews (e.g., training, self-audits)
Compliance rate changes
Number of facilities that have changed regulatory status
Number of facilities that eliminated, treated or reduced emissions or other
pollutants
Quantified environmental improvements (e.g., amount of emissions or
pollutants eliminated, treated or reduced)
D.
Which Data Collection Method Best Meets Your Needs?
Gathering the necessary data to evaluate compliance assistance activities is usually the most
difficult, time-consuming, and resource-intensive step in the evaluation process. It is essential that you
select the most appropriate data collection tool to meet the goals of your evaluation.
Data Collection Tools
Whether you conduct an anecdotal assessment or statistical study, there are a number of data
collection tools available. These tools include surveys (mail, fax, email, Internet, and phone) and
observed data (on-site revisits, pre/post tests, and reviews of self-reported data). Where possible, OECA
recommends using observed data as they are more objective than changes reported by survey participants.
In general, you can collect more detailed information using telephone surveys and on-site revisits than
using mail, fax, email, or online surveys, since they allow you to ask follow-up or clarifying questions.
-------
Section III: How to Plan and Design an Assessment
Page 17
Another consideration in selecting a tool is the expected response rate; Table 3-2 presents response rates
for surveys conducted as part of a set of federal pilot projects. OECA does not recommend using mail-
back comment cards (tear sheets); EPA and states found that these cards were potentially biased and
received very low response rates (usually around 2 percent).
Table 3-2. Pilot Project Response Rates
Measurement Method1
Pre-/Post-tests (1)
Phone survey (3)
Mail survey (7)
Email survey (1)
Workshop survey (1)
Revisits (3 inspections (2)
Inspections (2)
Range of Response Rates
97%
41% -100%
2% - 46%
100%
85%
100%
100%
Average Response Rate
97%
80%
17%
100%
85%
100%
100%
The number of projects on which the data are based is given in parentheses.
Additional considerations for each of the data collection tools are discussed below:
Pre-/Post-Test. Before conducting the compliance assistance activity (e.g., workshops,
training sessions), consider testing attendee knowledge of regulations you plan to cover.
At the end of the compliance assistance activity, retest the participants to determine
changes in understanding of the materials presented. Similarly, you can assess behavioral
practices at the facility before a workshop and practices reported in a follow-up survey to
identify changes made. Pre-/post-tests can also help you improve your compliance
assistance materials by revealing areas where key messages did not come across. Pre-
/post-tests conducted at workshops are considered part of the workshop and are exempt
from the Paperwork Reduction Act.
Telephone Survey. A telephone survey is a standard set of questions asked to potential
respondents over the telephone. These surveys, used alone or in combination with mail or
online surveys allow you to ask follow-up or clarifying questions, potentially resulting in
better data than a mailed survey. Telephone surveys work best if the list of potential
respondents is a manageable number (OECA recommends less than 50 respondents). To
reduce costs, some regions have hired college students to make the call-backs.
Mail/Email/Fax Survey. A mail, email, or fax survey is a set of questions sent to
potential respondents with a request that they voluntarily respond. These surveys enable
you to reach a large number of potential respondents, and may be the best option where
there are more than 50 recipients. However, mail/email/fax surveys can provide
ambiguous results, since it is not easy to immediately follow up and clarify unclear,
-------
Page 18
Section III: How to Plan and Design an Assessment
conflicting, or unexpected responses. Similarly, a limited level of detail is obtained, as
respondents will generally not spend the time to write long answers to open-ended
questions.
Online Survey. An online survey is a set of questions posted on a Web site or list server.
These surveys have the potential to reach a large number of respondents. For surveys on
Web sites, you can reach users that might otherwise be unknown to you. Many
respondents like online surveys because they can respond at their convenience and they
do not need to worry about losing a survey or mailing it back. As with mail surveys,
however, the online survey may provide only limited detail as respondents might not
want to spend time typing in a longer response. In addition, without follow-up, there is
potential for ambiguity or conflicting results, as with the mail survey. Another problem
with a simple online survey gauging the effectiveness of a Web site is that respondents
may be biased. For example, only respondents who found a Web site to be very useful
typically respond to the survey while those who did not find the site useful or found it
only marginally useful typically do not respond to the survey. To reduce nonresponse
bias you may want to advertise the surveys on relevant list serves to give non-regular user
an opportunity to respond.
On-Site Revisit. On-site revisits
involve returning to facilities that
previously received a compliance
assistance visit. Revisiting
facilities can provide excellent
data since you can use direct
observation to make assessments
and because facilities are likely
to spend the necessary time to
answer questions while you are
on site. In addition, the revisit
itself might spur additional
compliance assistance or
pollution prevention activities.
Revisits can be performed by
either compliance assistance staff
or by inspectors. For the best
results and highest level of
cooperation, this data gathering
should not be linked to enforcement initiatives. If revisits are performed by inspectors, it
is critical that the information collected for the compliance assistance evaluation is not
used to make the site a target for enforcement.
Data Reviews. Reviewing in-house data such as permits, permit applications, and TRI
forms can also help you assess changes in understanding and behavior. For example, if
your project sought to improve the quality of TRI Form R submissions or stormwater
permits, you can assess the quality of these documents before and after the training or
compare submissions from those receiving assistance and those who did not receive
assistance to gauge your progress.
IN DENVER, DIRECT OBSERVATION
EQUALS RELIABLE ASSESSMENTS
Region 8 delivered multimedia compliance
assistance, pollution prevention, and cost
savings tools to auto service and repair shops
in the Denver metropolitan area. Via on-site
revisits, Region 8 inspectors measured
behavioral changes (including changes in
compliance and beyond-compliance actions),
and also looked for changes in awareness and
understanding. By using direct observation,
the inspectors were able to make reliable
assessments and learn about the business (and
language) of auto service and repair.
-------
Section III: How to Plan and Design an Assessment Page 19
You can also consider a combination approach, such as using a mail and/or online survey
with a phone survey to selected participants or to reach non-respondents. Such an
approach enables you to potentially reach a large number of respondents, yet also collect
detailed information from selected participants. In addition, it enables you to ask follow-
up questions and clarify any unexpected results from the mail survey. In general, the
more resources you expend, the more responses and detail you will receive. Table 3-3
highlights the uses of the different tools, the resource considerations, average response
rates (for OECA pilot projects), and tips for lowering costs and improving response rates.
-------
Page 20
Section III: How to Plan and Design an Assessment
Table 3-3. Compliance Assistance Data Collection Tools
Method
When to Use
Resource Considerations
Range of
Response Rates1
Response Rate Considerations
Comment Cards
OECA strongly discourages using mail-back comment cards (tear sheets). EPA and states found that these cards were potentially biased and received very low response rates
(usually around 2 percent).
Pre-/Post-Tests
To measure
changes in
understanding
Does not require OMB approval if conducted
with workshop. Requires staff time to
administer and adds time before and after
workshops/presentations.
97%
Conduct tests immediately before and after compliance assistance
workshops/presentations when respondents are already assembled. Conduct
post-tests prior to the formal close of event, if possible.
Telephone
Survey
To obtain detailed
data.
Long-distance charges will apply if surveying
a large geographical area. Staff time to
administer telephone survey is higher than the
mail or online survey. Staff will need to be
trained to ensure survey consistency.
41-100%
Call respondents either early or late in the day to bypass secretarial screening.
Also consider using non-EPA employees to conduct the survey to improve
perception of anonymity.
Mail/Email/Fax
Survey
To reach many
respondents.
To obtain "yes/no"
answers.
Mailed surveys require postage costs for
sender and respondent. Reminder postcards
require additional postage. Email and fax
surveys can be limited by technical (e.g.;
computer hardware) capabilities.
2-46%
Potentially low response rate if used alone. Consider supplementing with
telephone surveys. Follow the Tailored Design Method (TDM)suggestions in
Section IV. (TDM had been shown to average a 65% or higher response rate.)
Online Survey
To evaluate
electronic services.
Can also be used
as an alternative to
a mail survey.
Up-front costs include posting the survey on
the Web site and programming electronic data
entry capabilities, if desired. Can eliminate
data entry costs associated with other surveys.
N/A - Total
population will
not be known.
Prompt Web users to respond to the survey by announcing it to the target
audience through list servers, newsletters, conferences, etc., and by making it
noticeable on the Web site. Online surveys exclude compliance assistance users
without access to Internet and can self-select favorable responses.
On-Site Revisits
To obtain detailed
or difficult-to-
collect data (e.g.;
environmental
changes)
Staff time is required to set up and administer
the survey to participants. Evaluators will also
incur travel costs to the facilities. Resource
constraints often limit the number of facilities
you can visit. Staff will need to be trained to
ensure survey consistency.
100%
Try to get a commitment for a revisit from the facility prior to the initial
assistance visit.
From OECA pilot projects.
-------
Section III: How to Plan and Design an Assessment
Page 21
Selecting the Proper Evaluation Method
In general, the evaluation method used should match the type and intensity of compliance
assistance activity. It makes more sense, for example, to use on-site revisits to follow up on previous
visits than it does to follow up on those who simply received a guidebook or called for telephone
assistance. Table 3-4 offers OECA's recommended follow-up methods for each type of compliance
assistance activity.
Table 3-4. OECA's Recommended Follow-up Methods by Assistance Activity
Compliance Assistance
Activity
On-site visits
Workshop/presentations
Compliance guides
distributed through
workshops
Compliance guides
distributed via mail
Compliance guides
distributed via the
Internet
Hotlines
Compliance assistance
centers - Internet
Preferred Follow-up Method
1. On-site revisit; or
2. Phone survey
1. Pre-/Post-tests; and
2a. Phone survey if <50 attendees; or
2b. Mail survey if >50 attendees
1. Phone survey if <50 attendees; or
2. Mail survey if >50 attendees
Mail survey
No follow-up for outcome measurement;
count number of guides downloaded only
Periodic user survey
Online survey
Conditions
1. Do at least 25% revisits; or
2. Call 100% of sites
1. Test all participants; and
2a. Phone all participants; or
2b. Use the Tailored Design Method
(TDM)9
1. Phone all participants; or
2. Use the TDM
No mail back cards
1. Maintain a list of mailed recipients
and use the TDM
No mail back cards
No mail back cards
Use phone surveys
Use a secured site that would eliminate
participants from taking the survey
multiple times
See Section IV of this guide for more information on using the Tailored Design Method.
-------
Page 22 Section III: How to Plan and Design an Assessment
E. Do You Need an ICR?
Congress enacted the Paperwork Reduction Act (PRA) to protect businesses, states, and the
general public from burdensome paperwork. The act requires federal agencies10 to obtain approval from
the Office of Management and Budget (OMB) prior to collecting substantially similar information from
more than nine nonfederal respondents. As defined by the PRA, a "collection of information" means
obtaining or soliciting information through identical questions, or reporting or recordkeeping
requirements. The PRA applies to both mandatory and voluntary data collection efforts; therefore, most
compliance assistance evaluations are subject to the PRA. Note, however, that the following actions do
not require OMB approval:
• Surveys handed out on-site immediately after a workshop, seminar, or meeting that ask
participants about the quality of the seminar (e.g., knowledge of speakers, usefulness of
handouts);11
• Pre-/post-tests conducted at a workshop, seminar, or meeting; and
• Attendance sign-in sheets at a meeting, workshop, or Web site.
Appendix C provides additional information to help you determine whether or not your evaluation falls
under the PRA.
Figure 3-1 should help you determine what next steps are necessary to proceed with your
evaluation. In general, when you ask the same set of questions (whether voluntary or regulatory) to more
than nine people, the survey will fall under the PRA12. OECA has, however, obtained an expedited
approval process for many compliance assistance surveys through its generic ICR. You can use the
generic ICR when your goal is to determine the effectiveness of compliance assistance activities on the
audience that receives the compliance assistance (e.g., participants at a workshop or users of a
compliance assistance tool). Note, however, that the generic ICR cannot be used when you plan to use a
10 The PRA applies to federal projects only. The act does not apply to states unless they are conducting a survey on
behalf of EPA. See Appendix B for more information.
11 Surveys to collect baseline data to assess awareness and understanding prior to the workshop may require OMB
approval. Consult Appendix C for specific scenarios.
12 Note that this requirement does not apply when contacting officials at federal facilities.
-------
Section III: How to Plan and Design an Assessment
Page 23
statistical approach to generalize the effectiveness of a compliance activity on an overall population. In
this case, you will have to develop a separate ICR for your evaluation.13
Key
O Dm!,im Modu
: Action Midi
niors d-nn 9 r^rfe
=mp)or,i<:c5!
Yes
- »^
i ''"I/""' " N
I "•' »"'•>•*••"»- ',
V ' • '"""- • /
Yes
r
No
Is nc
c" ICR! liWXi'i
Yes
E
Submit juiveyinstalment, cover memo and
bundenestiiiialie to CASPD for reviav.
C ASPD wil forwajd tte survey to OMB
tlwoughthe Office of Environmental
information. Allow at least 30 days for
OMB approval
jtAp. The KK
the C^ce ^
virOTTiierGS
id^«. Col -sctbn
i: hrap-: *V lit r En
fcnr.a
-rn the iPA
Figure 3-1. Do You Need An ICR?
13 OMB put this constraint on the ICR to give the public an opportunity to comment on survey methodology prior to
implementation. For guidance on developing an ICR, see EPA's ICR Handbook available at from the Office of
Environmental Information, Collection Strategies Division (CSD) found at
http://intranet.epa.gov/icrintra/download.html. CSD provides policy direction and oversight of Agency management
of regulatory information and manages the Agency's administration of the burden reduction provisions of the
Paperwork Reduction Act. CSD's OECA liaison will answer any questions you might have about whether an ICR is
needed, issues involved in preparing an ICR, or the ICR clearance process. Contact CSD by calling their general
number at (202) 566-1700 and asking for the ICR contact.
-------
Page 24 Section III: How to Plan and Design an Assessment
If your survey is covered under the generic ICR, you will still need to obtain clearance from OMB before
you distribute your survey. To obtain clearance, you need to submit a copy of the survey instrument,
cover memo, and burden estimate to EPA Headquarters, Compliance Assistance and Sector Programs
Division. Appendix B contains the Office of Compliance Guidance on the Need for Information
Collection Requests (ICRs) for the Evaluation of Compliance Assistance Activities. Appendix C contains
information on how to obtain clearance for regional compliance assistance evaluation surveys under the
generic ICR for compliance assistance.
-------
Section IV: How to Get the Most out of Your Survey Page 25
SECTION IV: HOW TO GET THE MOST OUT OF YOUR SURVEY
This section will help you design clear and effective questionnaires, techniques to implement
efficient surveys to provide high response rates, and organize your findings in a meaningful way. OECA
developed this guidance by consulting survey experts and analyzing the results of pilot projects. This
section draws on Don Dillman's14 work in designing and implementing surveys. OECA strongly
encourages you to follow these suggestions, which can help you maximize the effectiveness of your
evaluation efforts.
Section A discusses how to write good questions and how to construct the questionnaire and
Section B discusses how to effectively implement the survey. Section C discusses and defines basic data
analysis techniques. Section D discusses lessons learned from two regional projects. Appendix D for
samples of survey questions by outcome measure.
A. Questionnaire Design
The questionnaire development phase of your survey project is very important. Numerous
evaluation projects have failed to yield reliable information because the survey instrument has been
flawed (Dillman, 1999). A poorly constructed questionnaire can lead to increased nonresponse,
misinterpretation of questions (resulting in unreliable information), incomplete answers to questions, and
unanswered questions.
The most important aspect of questionnaire design is whether the questions are salient to the
respondent. That is, the respondent should feel the need to provide the information you are requesting.
Salience is created through writing good questions and organizing those questions effectively, as
discussed below.
14 Don Dillman is the Thomas S. Foley Distinguished Professor of Government and Public Policy in the Departments
of Sociology and Community and Rural Sociology, and Deputy Director for Research and Development in the
Social and Economic Sciences Research Center (SESRC) at Washington State University. He is recognized as one of
the leaders in survey research in United States and has over 35 years of experience in designing and conducting
survey research. See his publications at http://www.sesrc.wsu.edu/dillman/default.ASP
-------
Page 26 Section IV: How to Get the Most out of Your Survey
Tips on Writing Effective Questions
Some Questions You Should Ask About Each Question
To begin, once you have formulated a set of questions, you should review those questions to
determine if they are something that respondents will want to answer and if they will provide the
information that you need. Dillman (1999) suggests that you ask the following eight questions about each
survey question.
Is the question written so that each respondent can provide an answer? If your questions
provide a set of potential answers, make sure that all possible answers are covered. That is, no
respondent should look at the set of potential responses and not find an answer that fits their
situation.
To what extent do survey recipients already have an accurate, ready-made answer for the
question they are being asked? To ensure that the question is answered, you should make sure
that you are asking something that respondents can answer with relatively little effort. When
possible, you should avoid including questions that require respondents to look up information or
to make calculations.
Will the respondent be able to accurately recall past events and behaviors related to the
question? You should consider whether or not the information you are requesting is something
that the respondent can recall easily.
Will the respondent be willing to reveal this information? You should consider whether or not
the respondents will reveal the information you are asking for and how truthful those answers will
be. This is especially critical in evaluating information related to compliance with environmental
regulations.
Will the respondent feel some motivation to answer the question? Although you may find the
answer to a question interesting, you should consider whether or not the respondent would feel
the same way. Research has shown that the more salient that a question is to a respondent, the
more likely that respondent will answer the question.
Will the way in which the question is presented influence the answers that the respondents
may give? You should think about the answer categories that you have provided to the
respondent and consider whether or not those will have some influence on the answers provided.
The Principles of Writing Good Questions section below discusses some of these considerations.
Are the data being collected in more than one manner? Some survey efforts use more than
one mode of collecting information (e.g., mail and personal interviews). If you want the data
collected to be consistent across the modes, you need to make sure that the mode you use will not
influence the answers you get. The Principles of Writing Good Questions section below discusses
-------
Section IV: How to Get the Most out of Your Survey Page 27
some of these considerations.
Is it possible to change the wording or structure of a question? Although a question may be
poorly written and have a number of faults, it may not be open to change. For example, you may
want your data to be consistent with a previous survey effort and therefore you would need to use
the same question wording.
Open-Ended Versus Closed-Ended Questions
Questions can be divided into two basic categories: open-ended and closed-ended. For open-
ended questions, the surveyor has not provided a set of answers from which the respondent can choose.
Closed-ended questions, on the other hand, provide a set of answers from which the respondent can
choose. One of these question types is not necessarily better than the other in all survey situations.
Nevertheless, you may want to consider writing many of your questions as closed-ended since it is
relatively easier to construct reliable closed-ended questions than reliable open-ended ones.
Open-ended questions work well when they have unambiguous interpretation. For example, the
question "How old are you?" is an open-ended question if you simply supply a blank space for the
answer. You may also want to give the respondent free rein on answering a question. These may be
exploratory questions (e.g., "What environmental compliance issues do you see confronting your business
in the near future?") or you may be seeking feedback on a program you have instituted ("How might we
improve this seminar?").
Not all closed-ended questions are easy to construct. Dillman (1999) divides closed-ended
questions into "ordered" and "unordered" categories. Ordered closed-ended questions provide the
respondent with a set of ordered responses (e.g., like or dislike on a scale of one to five). In an ordered
question, the respondent must simply place herself on a scale. Unordered questions provide a set of
mutually exclusive categories that have no particular order (e.g., choosing among a set of statements that
best describes a respondent opinion or situation). In an unordered question, the answering task is much
more complicated since respondents must compare one answer to all other answers. Thus, respondents
may find unordered questions with many answer possibilities more difficult to answer than some open-
ended questions since more evaluation must be done.
-------
Page 28 Section IV: How to Get the Most out of Your Survey
Dillman 's Principles for Writing Good Questions
Dillman (1999) provides a set of 19 principles for writing good questions. We have highlighted
the key steps. You should refer to Dillman's book15 for the complete list and discussion of the rationale
behind each principle as well as insightful examples.
1. Choose simple rather than specialized words.
2. Use as few words as possible in posing the question.
3. Use complete sentences.
4. Avoid vague quantifiers when more precise ones can be used. For example, use
quantifiers such as "1-2 times per month" or "less then ten" rather than "rarely" or "just a
few."
5. Avoid requesting a specific answer to a question if the respondent may not be able
to provide such an answer easily. For example, it may be too specific if you provide a
blank space for the question "How many autos were repaired in your shop in 2001?"
However, if that same question supplied a set of ranged responses (e.g., "less than 100,"
"100-200," etc), then the question may be easier for a respondent to answer.
6. For scaled questions, use an equal number of positive (e.g., agree) and negative (e.g.,
disagree) categories. For example, the question "How useful did you find our training
seminar?" could be followed by four answer categories: "very helpful", "somewhat
helpful", "somewhat unhelpful", and "not helpful at all".
7. Place "neutral" categories (e.g., "neither agree nor disagree") in the middle of
scales, but place "undecided" categories at the end of scales.
8. For unordered closed-ended questions, make all comparisons (i.e., answers)
equivalent to avoid bias. For example, the question "What factors have led to
environmental issues for your industry?" that provides "Irresponsible chemical
management" and "Economic hardship" does not provide equivalent answers. The use of
"irresponsible" makes that answer a value judgement while the other answer is more
objective.
9. For agree/disagree type questions, mention both sides in question. For example, use
"To what extent do you agree or disagree with the following statements?" rather than
"To what extent do you agree with the following statements?"
15 Dillman, Don, 1999. Mail and Internet Surveys: The Tailored Design Method, John Wiley and Sons, Inc: New
York.
-------
Section IV: How to Get the Most out of Your Survey Page 29
10. Avoid "Check All That Apply" questions. Research into survey design has found that
respondents tend to answer check off "just enough" of the response options to form what
they consider to be a satisfactory answer, rather than completely evaluating each
response option.
11. Be sure that answer categories are mutually exclusive.
12. For questions that require the respondent to recall past events, ask a few simple
questions regarding the details surrounding the event to improve respondents'
recall. For example, you may want to know respondents' thoughts on a seminar. To
improve recall, ask some simple questions about the seminar ("Did you attend by
yourself or with others?", "Did you have to travel a long distance to get there?").
Research has shown that these techniques can assist respondents in recalling the event
more clearly.
13. If you are comparing the data you are collecting with other data (e.g., previous
survey efforts, census data), be sure that the wording allows that comparison to be
made. In other words, make sure you are asking the same questions as the original
survey. This is particularly important for projects where data from two surveys done at
different times (i.e., pre and post compliance assistance efforts) will be compared.
14. Avoid using double negatives.
15. Do not use double-barreled questions. For example, the questions "Did you find the
seminar and the materials we passed out afterwards helpful?" asks the respondent to
evaluate two things at once (the seminar and the materials) and to provide one answer for
both. It could be that some respondents found the seminar helpful, but not the materials.
16. If you have to ask objectionable questions, find a way to word the question to reduce
the impact of that question. For example, the question "Does your facility have
compliance problems?" can be interpreted as threatening, but "Has your facility ever had
trouble in complying with some regulatory requirements?" may be less threatening since
it reduces the forcefulness of the statement.
17. Do not ask respondents to make unnecessary calculations.
Ideas for Constructing Questionnaires
To maximize the effectiveness of your survey effort, you also need to design the questionnaire in
an effective manner. Simply writing down a set of questions and sending that out as a questionnaire may
result in invalid or misleading data. Questionnaire design is an integral part of any survey effort.
In designing your questionnaire you should have two primary objectives: (1) reduce the amount
of nonresponse and (2) reduce the amount of measurement error (Dillman, 1999). Nonresponse is when a
-------
Page 30 Section IV: How to Get the Most out of Your Survey
potential respondent either makes no attempt to respond to any of the questions or leaves some questions
unanswered. A well-designed questionnaire will encourage potential respondents to begin answering the
questions and will ensure that no questions are missed. Measurement error is when respondents provide
incorrect answers. It can occur when a respondent is confused about how to answer a question and can be
caused by a poorly designed questionnaire.
Dillman 's Thoughts on Question Ordering
Question ordering for surveys is a complex and often-discussed subject among survey designers.
Dillman (1999) has provided the following six principles that you can use in ordering questions for your
survey. One thing to consider when applying these principles is that they may contradict one another.
That is, one principle may imply that a particular question be placed near the end, while another principle
suggests that the same question be placed near the beginning of the questionnaire. In designing your
questionnaire, you will need to weigh these competing influences.
Begin with questions that are the most salient to the respondent and work towards those
that are the least salient. The respondent should have some association with the subject you are
interested in, or that person would not be in the target population. Additionally, the cover letter
that accompanies your survey should make a case that the subject matter is important. This means
that respondents will expect to see and be more motivated by questionnaires that begin with
questions that interest them. One implication of this is that you should not begin a questionnaire
by asking a set of demographic questions.
Group similar sets of questions together. Respondents find it easier to answer similar set of
questions when they are grouped together rather than scattered throughout the survey.
Place potentially objectionable questions near the end of the survey. An objectionable
question may cause a respondent to discontinue the survey. This is less likely when such a
question is at the end of the survey.
Order questions in a manner that will be logical to the survey respondent. You should
consider how the respondent sees the subject matter and order the questions accordingly.
Group questions together that have similar components. For example, if you are asking a
series of questions that require the respondent to rate their agreement or disagreement on a scale
of one to five, then grouping these questions together may make sense. The reason is that the
respondent can remain in the mindset appropriate for answering those types of questions.
Choose the first question carefully. The first question on a survey is the most important
-------
Section IV: How to Get the Most out of Your Survey Page 31
question in determining whether the survey will be returned or thrown away (Dillman, 1999).16
Dillman suggests the following in choosing an effective first question:
(1) The question should apply to everyone that is responding to the questionnaire.
(2) The question should be simple, requiring only a few seconds to read and answer.
(3) The question should be something the respondent feels the need to answer.
Dillman's Thoughts on Constructing the Questionnaire
Once you have developed a set of questions and decided how to order them, you need to compile
them into a questionnaire that will encourage the respondents to return the questionnaire. First, you need
to create a questionnaire that creates the same stimulus for each respondent. Second, you need to
construct the questionnaire in a way that focuses the respondent on answering the questions rather than on
deciphering the questionnaire. This is often an overlooked aspect of survey design, but a poorly designed
questionnaire can lead to high levels of both nonresponse and measurement error. Dillman has developed
a set of 34 principles for constructing questionnaires. This guide does not repeat each of these, but rather
includes a subset of 18 of those principles that EPA feels are most useful in conducting performance
measurement surveys. These principles deal with both the visual design of the questionnaire and the
placement of various pieces of information.
One common idea about improving response rates is that if the questionnaire is shorter, your
response rate should be higher. This is true to some degree, but it will depend on how the questionnaire
was shortened. Certainly, removing irrelevant and longer questions effectively reduces questionnaire
length. On the other hand, reducing the font size or reducing spacing between questions may be
counterproductive. These methods of reducing length may frustrate or confuse the respondent. Focus on
designing the questionnaire effectively and logically, rather than on shortening the length through visual
manipulation. Many of Dillman's principles will actually increase the length of a questionnaire by asking
you to organize a question so that more vertical space is required for questions. His reasons for this,
however, are based on research that has demonstrated that visual factors are more important than numbers
of pages for influencing response. If you need to reduce the length of your questionnaire, focus first on
removing unnecessary questions and then on the types of answers you are requesting (e.g., open-ended
16 This is not to say that the first question is the most important/actor in determining nonresponse rates, but is simply
the most important of all the questions.
-------
Page 32 Section IV: How to Get the Most out of Your Survey
questions usually require more space than closed-ended ones).
Table 4-1 presents the 18 principles that EPA feels are the most important in developing
performance measurement surveys. They may seem somewhat repetitive, but the purpose is to provide a
set of principles that cover all general situations, while also dealing with some specific ideas. As with
some of the other principles above, satisfying all of these at once may not be possible since there are some
contradictions.
Table 4-1. Subset of Dillman's Principles of Questionnaire Construction
No.
Principle
Write each question so that respondents do not need to re-read the question to know how to
respond.
Place instructions exactly where the information is needed and not at the beginning of the
questionnaire.
Use "item-in-a-series" lists to organize items with the same response categories, but be careful. An
example of an "item-in-a-series" list would be asking respondents to rate, on a scale of one to five, a set
of statements and then listing those statements in one column on the left with columns of ones through
fives on the right. This is an effective manner of organizing questionnaires, but be aware that respondents
tend to view these types of lists as comparative in nature and will respond to the set of items rather than to
each question individually. That is, the answer to one question in the list will be made relative to all other
questions in the list.
Ask one question at a time. For example, do not ask respondents to rate a presentation on a scale of one
to five and then provide thoughts on how to improve the presentation.
Use similarity and diversity to identify groupings of visual elements. Questions should all look the
same, similar response categories should all look the same, and instructions should all look the same.
Additionally, you should make each group look different from one another (e.g., questions: bold; response
categories: regular font; and instructions: bold italics).
Maintain simplicity, regularity, and symmetry to simplify the response task
Number questions consecutively and simply, from beginning to end. That is, if you divide the
questionnaire into sections, do not restart the numbering or use numbers such as "5.1."
Leave more blank space between questions than between the subcomponents of questions.
Use dark print for questions and light print for answers.
10
Use question-specific instructions inside the question number and not as free-standing entities.
11
Separate optional or occasionally needed instructions from the question by a font change.
12
Do not place instructions in a separate book or in a separate section.
-------
Section IV: How to Get the Most out of Your Survey Page 33
No.
Principle
13
List answer categories vertically instead of horizontally, where possible. For example:
How clear was the presentation of the material at the training seminar?
D Very clear
D Somewhat clear
D Somewhat unclear
D Very unclear
14
Place answer spaces consistently to either the right or the left of answer category labels. For
example, use either "n Agree" or "Agree n" and use the same order throughout.
15
Use numbers and simple answer boxes for recording answers. For questions that require a numeric
answer, provide a blank line or empty box and provide a label for that answer (e.g., " pounds per
month"). For closed-ended questions that provide pre-set answers, use a box such as this: n.
16
Avoid double- or triple-banking of answer categories. For example, if you are providing 15 possible
choices for a specific questions, do not arrange those in three columns of five. Either arrange them in one
vertical column or reduce the number of choices.
17
Maintain the direction of scales throughout the questionnaire. That is, do not switch from displaying
five-point scales as "54321" to displaying them as "1 2 3 4 5." Additionally, do not switch from having
5 as the "most positive" answer to having 1 as the "most positive" answer.
18
Use shorter lines to prevent some potentially key words from being missed. Visually, shorter lines
tend to be read more completely than longer lines.
Dillman, Don, 1999. Mail and Internet Surveys: The Tailored Design Method, John Wiley and Sons, Inc: New
York, Chapters.
B. Survey Implementation-The Tailored Design Method
This section discusses the Tailored Design Method (TDM) for implementing surveys. Dillman
developed the TDM, which he describes in detail in his book Mail and Internet Surveys: The Tailored
Design Method (John Wiley and Sons, Inc., 1999). The method is based on the principles of sociology
and takes into account how people react and respond to questionnaires. This section first briefly discusses
the five elements of the TDM and then discusses one of those elements, the stages of contact with
potential respondents, in more detail. Appendix E contains sample letters of each stage of Dillman's
contact sequence that were used for the OC Metal Finishing program discussed in Section VI.
The Elements of the Tailored Design Method
Dillman's TDM comprises five distinct elements. Taken as a whole, these elements have been
-------
Page 34 Section IV: How to Get the Most out of Your Survey
shown to achieve good response rates for mailed and other self-administered surveys. The five elements
are:
Element 1—Develop respondent-friendly questionnaires;
Element 2—Implement a sequence of five contacts with the respondents;
Element 3—Provide return envelopes with first-class stamps;
Element 4—Develop personalized correspondence; and
Element 5—Provide token prepaid financial incentives.
The first element (respondent-friendly questionnaires) was discussed in great detail in Section A. Element
2 (the contact sequence) is discussed in a separate section below. This section briefly discusses each of
the other three elements.
Provide return envelopes with first-class stamps
Dillman suggests that the return envelopes that you provide for the respondents should contain
first-class stamps. This personalizes the response process since stamps are less formal than metered
envelopes. Research has shown that stamped envelopes are more effective than metered envelopes. Some
respondents may very well take the stamps and not return the survey. Dillman, however, notes that
respondents tend to be motivated by trust. In placing stamps on the return envelope you have said to the
respondent, "we trust you will return this, but if not, here are some stamps."
Develop personalized correspondence
Surveys that look and feel like mass-mailings will tend to be treated like mass-mailings (i.e.,
many potential respondents will discard them). Dillman suggests that an effective method of avoiding this
is to personalize each contact with the potential respondents. This includes signing each letter with an ink
that differs in color from the letter's font color (e.g., a blue pen), using names in the salutation (e.g., Dear
Ms. Smith), and writing letters using a familiar but businesslike tone. However, avoid over-personalizing
correspondence since this can sound insincere.17
17 For example, "This new program will affect your shop, Smith's Auto Repair, and may result in a change in the
way that you conduct exhaust repairs" may be too personalized by using the name and the type of business.
-------
Section IV: How to Get the Most out of Your Survey Page 35
Provide token prepaid financial rewards
Dillman suggests that the use of token prepaid financial rewards can significantly increase
response rates. He cites several studies that show that, by providing prepaid financial rewards, you can
increase response rates by invoking a sense of trust in the respondent. However, in many cases financial
incentives for returned surveys are prohibited by law for government-sponsored surveys, therefore you
may need to think of alternative incentives.
EPA has authority under various statutes to conduct, or provide financial assistance to conduct,
surveys. Examples of statutes authorizing surveys include Section 103 of the Clean Air Act, Section 104
of the Clean Water Act, and Section 8001 of the Solid Waste Disposal Act. These statutes authorize
surveys but do not provide exemptions from the Paperwork Reduction Act; the Agency must comply with
OMB procedures relating to information collection when it conducts a survey via staffer contractor or
funds a survey under a grant or cooperative agreement.
The Agency may use appropriated funds for survey response incentives for EPA sponsored
surveys. Particular care must be given to the selection of survey incentives for EPA sponsored surveys;
the Agency can not endorse or appear to be endorsing a specific product or company. For example, the
Agency can offer a Personal Digital Assistant (PDA) as an incentive for completing a survey but should
not advertise a specific PDA brand in its survey outreach efforts.
The costs for survey incentives may be allowable under grants and cooperative agreements. In
situations where surveys are administered by a third party under a grant or cooperative agreement, the
EPA can offer equipment or other survey incentives to its grantees as a form of in kind assistance. In
cases where survey incentives are purchased and provided by an assistance agreement recipient, the
appearance of Agency endorsement of a particular product or company is no longer a concern. For
example, the Compliance Assistance Centers recipients have offered survey respondents the opportunity
to be entered into a drawing for an Amazon.com gift certificate upon completion of their surveys. While
this incentive is appropriate for the recipient to purchase, it would not be an appropriate incentive for
EPA to offer. Be sure to consult with EPA's Office of General Counsel on any planned survey
incentive to ascertain whether it can be offered.
-------
Page 36 Section IV: How to Get the Most out of Your Survey
Survey Implementation: Dillman's Five-Stage Contact Sequence for Mailed Questionnaires
Dillman (1999) proposes using a five-stage contact sequence to implement mailed surveys.18 He
has found that this five-step process increases response rates by 20 to 40 percentage points over a single
point of contact (e.g., a questionnaire with a cover letter only). Dillman advocates this staged process for
two main reasons. First, the multiple stages act as reminders to those that would like to respond, but who
have forgotten to do so. Second, the multiple stages offer the researcher several opportunities to convince
a nonresponder to reply. The idea behind the staged process is to provide a new stimulus with each
successive contact with the potential respondents. Using anew stimulus instead of simply resending
previous materials is more effective, since people tend to discount repeated stimuli. The five stages of
contact, along with their recommended timing, are:19
1) First contact—Prenotice letter;
2) Second contact—The questionnaire (4-7 days after the prenotice letter);
3) Third contact—Reminder postcard (one week after the original questionnaire);
4) Fourth contact—The first replacement questionnaire (two weeks after the reminder
postcard); and
5) Fifth contact—Sending a final questionnaire through a different delivery mechanism
(four weeks after the replacement questionnaire).
Below is a detailed discussion of each of the five steps.
You should develop the materials for the first three stages at the same time. This will reduce
logistical problems in administering the survey and allow you to develop a more systematic sequence.
You can then wait until after you have mailed the postcards to develop the materials for the fourth and
fifth stages. You may receive some information from respondents in the first three stages that will assist
18 Dillman proposes financial incentives. However, since these are unlikely for government surveys, such incentives
are not discussed in this section.
19
See Appendix E for sample letters for each stage.
-------
Section IV: How to Get the Most out of Your Survey Page 37
you in customizing these final two contacts.20'21
If you want to maximize the likelihood that your surveys are returned, you need to consider the
schedule of your potential respondents. First, try to determine when potential respondents are likely to be
busy and avoid those times. For example, mailing questionnaires to farmers at harvest time will likely
result in significant nonresponse. Second, try to determine when your survey will be most salient to the
respondents. For example, it is better to mail a survey on auto repair shop procedures for fixing air
conditioners in the summer than in the winter. Finally, avoid mailing the survey so that it arrives between
Thanksgiving and Christmas since this is always a busy time for people.
First Contact—Prenotice Letter
The prenotice letter tells respondents that they will be receiving a survey and that you would
appreciate their assistance by completing and returning it. The letter should be brief (one page) and
should not raise any concerns for the respondent. For example, avoid long discussions of confidentiality.
The letter should contain each of the following components:
• The date;
• The potential respondent's address (i.e., an inside address);
• A first paragraph that states the potential respondent will be receiving a survey and who
is sponsoring the survey;
• A paragraph that explains the subject of the survey;
• A paragraph that explains why the survey is useful and what the results will be used for;
• A thank you phrased in sincere terms; and
• A real signature.
20 For example, you may find that your sampling frame included a number of out-of-scope facilities. This may be
something to address in subsequent contacts.
21 If you need OMB approval for your questionnaires, it may be more efficient to include all five letters in the initial
approval request.
-------
Page 38 Section IV: How to Get the Most out of Your Survey
Finally, print the letter on the letterhead of the sponsoring Agency and mail it first class. Time the letter
so that it arrives only a few days ahead of your questionnaire.
Second Contact—The Questionnaire
Your second contact will include three items: a cover letter, the questionnaire, and a stamped
return envelope. Earlier sections discussed the questionnaire and the use of stamped return envelopes, so
this section discusses the cover letter for this mailing and also provides some tips on this stage of contact.
The cover letter for this mailing will differ from the prenotice letter by providing more detailed
information. Nevertheless, keep the cover letter to one page since longer letters will only raise concerns
among potential respondents. Your cover letter should include the following components:
• The date.
• The potential respondent's address (i.e., an inside address).
• A first paragraph that explains why the respondent is receiving the questionnaire. It
should explain in general terms the subject matter of the survey. Begin the paragraph
with an expression such as "I am writing to ask for your assistance in a study..."
• A paragraph that explains how and why the respondent was selected to receive a
questionnaire. This paragraph should explain who the target population is and how
members were selected from that group (e.g., randomly selected).
• A paragraph that explains the usefulness of the survey. Specifically, you should discuss
how you will use the results. If you can relate the usefulness of the results to the
respondent, do so. For example, for developing a compliance assistance program, include
a statement such as "We will use the results of this study to develop a program that best
fits your needs."22
• A paragraph that discusses confidentiality. Keep this paragraph simple and to the point. A
long discussion about confidentiality may raise concerns among some respondents. Some
respondents may not be convinced about confidentiality of the responses, but you cannot
direct your letter solely at those members of the population. In a subsequent paragraph,
provide a phone number where additional questions can be asked.
• A paragraph that indicates you are willing to answer any questions regarding the study,
the questionnaire, or confidentiality. Provide a phone number and the hours when
someone will be there to answer questions.
22 The example statement is very general; strive to be more precise with your own statements.
-------
Section IV: How to Get the Most out of Your Survey Page 39
• A sentence thanking the respondent for taking the time to complete the survey.
• A real signature.
As with the prenotice letter, print this letter on the sponsoring Agency's letterhead.
Finally, below are some additional tips on the second contact:
• Use questionnaire identification numbers—Each questionnaire should have unique
identification (ID) number. Print the number in the upper right corner of the
questionnaire cover page. This will assist you in tracking who has responded.
• Mail the questionnaire packet using first-class postage—Although this is more expensive
than other bulk means of delivery, Dillman's experience is that the benefits of using first
class far outweigh the costs.
• Review postal procedures—The U.S. Post Office changes postal procedures from time to
time. Before sending out a large mailing, you should review the most current postal
procedures and inform the local postal office of the size of your mailing. They may be
able to better accommodate you and provide some advice to ensure a successful mail-out.
• Put some thought into assembling the package—This can often be overlooked. All items
in the package should come out at once when the respondent opens the packet. Try
different ways of packaging the components and then ask someone who does not know
what to expect inside the envelope to open and remove the items. Also avoid using paper
clips to hold the materials together since they can jam postal sorting machines.
Third Contact—Reminder Postcard
This stage involves the final remainder that is sent to all members of the sample. One important
thing to remember about this stage is that using a postcard is not simply to save resources but to provide a
new stimulus for the potential respondents. The primary purpose of the postcard is to get responses from
those individuals who want to return the survey, but have yet to do so (e.g., have forgotten about or lost
the survey). The postcard will have little impact on "hard-line" nonrespondents. The fourth and fifth
contacts are directed at them.
Your postcard should contain five items:
-------
Page 40 Section IV: How to Get the Most out of Your Survey
• The date;
• A first paragraph that explains a survey was sent to them, when it was sent, and what the
survey concerned;
• A second paragraph that (a) thanks the respondent if they have already completed and
returned the survey, (b) encourages the respondent to complete and return the survey if
they have not done so already, and (c) a reiteration of how important each respondent is
to the study;
• A final paragraph that gives a phone number where they can request a replacement
questionnaire; and
• A signature.
The postcard should use the same name and address as the previous mailings. You do not need a
salutation on the text side of the card.
Fourth Contact—The First Replacement Questionnaire
This mailing will be similar to the second contact, but make your cover letter more insistent and
send this only to nonrespondents. Include new copies of the questionnaire and stamped return envelope.
Simply resending the original questionnaire packet and stamping "second notice" on the materials will be
ineffective. Those that did not respond to the initial request will probably not complete an exact copy. In
this stage, your cover letter must convey to the nonrespondents the importance of their response.
Your cover letter should include the following information:
• The date.
• The potential respondent's address (i.e., an inside address).
• A first paragraph that states you sent them a questionnaire, how long ago it was sent,
what the questionnaire concerned, and that you have not heard from them. Be direct and
sound as if you are singling them out. Although this may sound confrontational, it is an
effective means of obtaining responses. This first paragraph must get and hold their
attention to read the remainder of the letter. Remember that the people receiving this
letter have not yet responded and bold statements may be needed to get their responses.
-------
Section IV: How to Get the Most out of Your Survey Page 41
• A second paragraph that explains that others have responded. The idea here is make it
known that it is okay to respond since others have done so. That is, their response is not
being singled out, just their participation.
• A third paragraph that reiterates how important their response is to the study. Convey that
their response will add valuable information to the study.
• A fourth paragraph that gives them a way out of future mailings if they are not part of the
target population. For example, ask them to return the survey and state briefly why they
are not eligible.
• A fifth paragraph that reiterates your commitment to maintaining confidentiality.
Nonrespondents at this stage may have this concern, so assure them that their responses
are confidential.
• A sixth paragraph that conveys an understanding that some people may not want to
complete the survey. Ask them to return the questionnaire with a brief note on the front
explaining why they do not want to fill it out, or to simply return it blank. In return, offer
to remove them from future mailings. The information you obtain here may help you
understand the nature of your nonrespondents.23
• A real signature.
• A postscript that provides the phone number to call with questions.
As with the previous letters, print the letter on the letterhead of the sponsoring Agency.
Fifth Contact—Sending A Final Questionnaire Through A Special Delivery Mechanism
The final contact provides a new stimulus by using a special delivery mechanism (e.g., certified
mail) or a different method (e.g., telephone call) to get the questionnaire into the hands of
nonrespondents. This mailing will contain a new cover letter and new copies of the questionnaire and
stamped return envelope.
Once again, revise the cover letter to reflect the stage of contact. Specifically, your cover letter
should include:
• The date.
23 You can also combine the fourth and sixth paragraph, especially if the remaining number of nonrespondents is
small.
-------
Page 42 Section IV: How to Get the Most out of Your Survey
• The potential respondent's address (i.e., an inside address).
• A first paragraph that states you have been sending them materials and what those
materials have concerned (i.e., a study and the subject matter).
• A second paragraph that states the usefulness of the study.
• A third paragraph that explains "time is running out" for them to respond.
• A fourth paragraph that explains you have sent these materials via special mail service
(state which one specifically) to ensure that they receive the material because all
responses are important in producing reliable results.
• A fifth paragraph that explains the confidentiality of the survey.
• A sixth paragraph that thanks them for their participation. This should be more than one
sentence.
• A real signature.
Print the letter on the sponsoring Agency's letterhead.
Given the variety of delivery options, consider which one will work best for you. Among your
choices are:
• U.S. Postal Service certified mail;24
• U.S. Postal Service priority mail; and
• Courier services (e.g., Federal Express, United Parcel Service).
If you have several items to mail, contact the service you will use and inquire about any procedures you
should follow to ensure timely delivery.
Tracking Techniques
Survey tracking is a valuable part of your data collection effort. You can track surveys both in
regards to the population to which they apply and the time frame in which they are mailed and returned.
Population tracking can be helpful when sending surveys that are to be returned anonymously. By using
general codes printed on the survey, you can track the returned information by strata, such as state or type
1 Use this option only if you can ensure that someone will be there to sign for the package.
-------
Section IV: How to Get the Most out of Your Survey Page 43
of facility, without requiring the respondent to identify themselves through name or address. Mailout and
delivery tracking is used to track the progress of survey mailout and response activities. This information
can help determine the number of respondents from each step of the TDM contact sequence, as well as the
associated return rates. You can develop simple tracking systems using commercially available
spreadsheet or database programs. Keeping tracking information in an electronic media such as a
spreadsheet or database helps simplify analysis.
Data Entry and Handling
Once surveys are returned and logged into the tracking system, enter the data into some type of
electronic media. If the data are to be analyzed in multiple ways, the best option is to use a database
application where querying the data is relatively simple. If you will conduct statistical analysis on the
data, spreadsheets are effective. In most cases, data can easily be converted from a spreadsheet to a
database and back again if necessary.
Follow-up and Clarification
Some survey recipients may not completely fill out their forms or may provide contradicting
information. Other respondents may not write clearly. In these cases, if the survey was not anonymous,
you may need to follow up with the respondent to clarify the answers. Follow-up is most successful when
you can contact the respondent via telephone. In these calls, be as courteous and brief as possible to keep
the additional burden to the respondent to a minimum.
C. Where Do We Go From Here?
This section provides some basic information on analyzing the data that you have collected.
Following are definitions that will help you understand some basic analytical techniques. We then discuss
some basic analytical techniques, separating them into descriptive techniques and measures of association
and comparison. Appendix F expands on these concepts and presents advanced analysis techniques.
The following subsections provide details on how to implement a number of statistical techniques
as well as what those techniques will tell you. Below are some definitions that will help you understand
-------
Page 44 Section IV: How to Get the Most out of Your Survey
the discussions that follow.
Variable—A variable is some measure of the characteristics of your sample or population.
Examples of variables are the pounds of chemicals used by each facility, the compliance status of
a facility, and the answers to survey questions. Variables are what statistics analyze.
Limited Variables—A limited variable is one that can only take on values within a certain range.
For example, when coding a yes/no question, you usually code "yes" as 1 and "no" as zero, thus
limiting the variable to two values. A question that asks respondents to rate satisfaction on a scale
of one to five is also a limited variable. The opposite of a limited variable is an "unlimited" one,
or a variable that has no constraint on the range of values. The pounds of chemicals used by a
company could be considered an "unlimited" variable.25 In general, the term "unlimited" is not
used to describe the variable; rather, it implies that the variable is not considered limited.
Continuous Versus Discrete Variables—These are ways of describing the potential values that
variables can take on within the feasible range of values, where the feasible range of values is
defined by the variable's limits. A discrete variable has specific values, usually integers, that it
can take on. Examples of a discrete variable are yes/no questions and questions that ask
respondents to respond on a five-point scale, since the numerical answer is limited to a few
values. Although many discrete variables are also limited (see above), some are not. For example,
a question that asks how many pieces of a certain type of equipment that a facility has in use is
discrete, but not limited: (i.e., the variable can only take on integer values, but there may be no
limit on the number). A continuous variable can, theoretically, take on any numerical value. In
general, if the variable can take on decimal values (i.e., non integer values), then it is continuous.
Scaled Variable—A scaled variable is a discrete variable that has been given a set of values that
correspond to qualitative criteria. For example, a question that asks respondents to rate their
satisfaction with something on a "scale" of one to five where each value has a defined level of
satisfaction is a scaled variable.
Statistical analysis procedures are generally separated into either descriptive techniques or
inferential techniques. This section discusses some basic descriptive techniques you can use to describe
your data. The purpose of descriptive statistics is to provide a feel for the nature of the data.
Frequency Distributions
A frequency distribution summarizes of how the data are distributed across various numeric
values for certain characteristics or factors in the data. The distribution can tell you where the data are
concentrated and what values are uncommon in the data. Below is an example simple distribution for the
size of facilities responding to a surveys:
25 Granted, firms cannot use less than zero pounds of a chemical, which effectively "limits" this variable. However,
limits such as this are not generally considered limits unless a large number of facilities use zero pounds.
-------
Section IV: How to Get the Most out of Your Survey
Page 45
Table 4-2. Example Frequency Distribution
Number of
Employees
1 -100
101-200
201-300
301-400
TOTAL
Frequency
15
56
34
12
117
Relative
Frequency
The table shows that the largest number of facili
percent employed 200 or less.
This example contains the four standard
• Frp.aii.p.nc.v — The number of ent
12.8%
E
rm n"'
t*7..
z^) ]
10.2
A nn~i
f~\ (—
on
OU
;xes ei
1 c;
1(J
elgm
/o
Cumulative
Frequency
mploym^nt Distri
Cumulative
Relative
Frequency
jution12-8"/0
/I OU.
/o
/o
to/
nployed betwee
gnt*
3 of af
requeT
105
117
nlOl
icy dis
89.
100
—i /i /
7%
7%
0%
and 20
'.nutH-i1
0 peoj
MI:
le and that 60. 7
II
1 -,100+ 10J6-200 201-300 301-400
Figure 4-1. Example Chart of Frequency
Relative frequency—The percentage of the total repreJJisteibutthHifrequency for that
category;
Cumulative frequency—'
frequencies for categoric s
iThe sum total ofe Pre^cf Bt Qnd the
3 that have been listed above it; and
• Cumulative Relative Fre,
category and the relative
Although you do not have to calculate
each of these for your data, you
should always report the frequency.
The "cumulative" measures have little
meaning when the categories are
nonnumeric or unordered (e.g.,
counties in a state). For the most part,
your data and informational needs
will dictate what you calculate.
mency
been'.
lative frequency for each
listed above it.
1-100
101-200
201-300
301-400
Figure 4-2. Example Pie Chart for Frequency
Distribution
' In this case, it is a range of employees.
-------
Page 46 Section IV: How to Get the Most out of Your Survey
You can also use charts rather than tables to represent frequency distributions. The bar chart in
Figure 4-1 represents the same employment distribution as is shown in Table 4-2. Furthermore, there are
numerous charts that one can use to convey data. For example, the employment distribution data can also
be represented by a pie chart as shown in Figure 4-2.
Both the two charts and the table provide the same information (i.e., the relative size of each
employment class), but do so in slightly different ways. For the most part, which one to use falls to
personal preference. You should remember, however, that the table representation provides more
information and may be more suitable for a report. On the other hand, charts tend to work better in
presentations.
For numeric data, there is no standard method of defining the ranges that form the categories
(e.g., the 1 - 100, 101 - 200, 201 - 300, and 301 - 400 employee categories). In some cases, the categories
can be based on well-defined groupings (e.g., counties within a state). In other cases, the categories are
more difficult to define. The example uses equal-sized categories (100 employees per category). Real-
world situations are rarely this convenient. For example, in the case study in Section VI of the guide, we
divided the metal finishers into two unequal employment size classes: less than or equal to 10 employees
and greater than 10 employees. In short, you may need to formulate your distribution a few times before
you determine a set of classes that provides the information that you want.
Charts are also useful for presenting information on a number of questions simultaneously while
also providing comparative information for those questions. Consider the following set of questions
where the usefulness of a set of resources are evaluated by respondents.
Q2. Please indicate the usefulness of these activities. Rate each using the
following scale: 4-"very useful"; 3-"somewhat useful", 2-"somewhat not
useful", l-"not useful at all".
EPA Presentation for colleges and universities 1234
Other EPA presentations for broader audiences 1234
EPA audit protocols 1234
EPA lab guide 1234
Frequently Asked Questions Featured on EPA web site 1234
Other EPA website information 1234
-------
Section IV: How to Get the Most out of Your Survey
Page 47
EPA responses to phone calls/hotlines
1234
Questions of this sort provide information on the usefulness of individual resources, but also provide
information on the relative usefulness of the various resources. Figure 4-3 provides an efficient means of
providing both levels of information (individual and relative usefulness) in one chart.
100%^
80% -
60% -
40% -
20% -
0%
Unot useful at all 4
Hsomew hat not
useful 3
nsomew hat useful 2
nvery useful 1
CD
CL
i_
CD
o
O
o
o
o
2
CL
-t-*
T3
CD
;g
'^
O
&
CD
O
<
CL
LU
CD
cc
O
Figure 4-3. Example of Answer Distribution Chart
Measures of Central Tendency-Mean. Median, and Mode
A measure of central tendency attempts to provide a single-value description of where the values
in the data are centered. In other words, if we were to pull a number from the distribution, what value is it
likely to be closest to. The most common are:
Mean—The average value of the data;
-------
Page 48 Section IV: How to Get the Most out of Your Survey
• Median—The value that has half the values above it and the other half below it; and
• Mode—The most common value in the data, or the value that appears most frequently.
Each has strengths and weaknesses depending on the nature of the data.
The mean is almost always calculated and is generally a good measure of central tendency,
especially when the data are not highly skewed toward either low or high values in the data. The mean,
however, can be easily influenced by one or two outlier observations (i.e., observations that are either
very high or very low compared the rest of the distribution), thus skewing the estimated mean toward the
outliers. Means do not work well when the answer to a survey question is not numeric (i.e., a qualitative
answer).
The median is also almost always calculated for a set of data.27 Additionally, the median generally
overcomes the shortcomings of the mean (i.e., it works well when data are highly skewed since it is not
influenced by outliers). The median, however, does not easily lend itself to further statistical analysis,
thus limiting its use beyond descriptive purposes.
The mode is the value that appears most often in the data. The mode is not used as often as the
mean and the median and is relevant when the data are known, discrete values. For example, a survey
question that allows respondents five possible responses would be well described by the mode. Variables
that are more continuous, such as pounds of chemicals used, are not well described by the mode since the
data could, theoretically, take on different values for each respondent.28
These three measures are not substitutes for one another, but are three different measures that can
describe the data differently. Therefore, you could calculate all of them or only one or two.
Measure of Variability
A measure of variability tells you the extent to which the data values differ from one another. In
some sense, the frequency distribution provides this type of information. The measures described here,
however, summarize variability using a single value rather than a set of frequencies for different data
27 Most spreadsheet programs contain a median function.
28 One exception to this, however, is when a number of respondents report the same value. For example, in a sample
of 100 entities, 20 may state that they use zero pounds of a certain chemical, with the remaining 80 reporting 80
different values. In this case, the modal value of "zero" is a useful piece of information.
-------
Section IV: How to Get the Most out of Your Survey Page 49
ranges. For simple analyses, there are two measures of variability that you should be concerned with: the
range and the variance.29
The range is simply the difference between the largest and smallest value in the data. The range is
useful when you want to demonstrate that respondents reported either a wide range or a narrow range of
values for certain variables.
The variance and the standard deviation30 measure the extent to which the values in the data differ
from the mean value. You can calculate the variance of a sample by summing the squared differences
between each observation and the mean value and then dividing by the number of observations minus
one. Most spreadsheet programs contain a variance function. Larger variances indicate the data are more
disperse.
You can also use the standard deviation in situations when there is a normal distribution.31 If you
can assume that the population from which the data were drawn are normally distributed, then the
following rules hold:
• 68 percent of all observations are within one standard deviation of the mean;32
• 95 percent of all observations are within two standard deviations of the mean; and
• 99 percent of all observations are within three standard deviations of the mean.
These rules allow you to infer the overall distribution of the population.
Two Other Useful Descriptive Measures
Finally, there are two other useful descriptive measures: minimum and maximum values. These
values provide a useful estimate of the smallest and largest values for the underlying population.
29 In actuality, there are three measures. The third, which the standard deviation, is derived from the variance.
30 The standard deviation is simply the square root of the variance.
31 A normal distribution is also referred to as a bell curve. For normal distributions, the mean and median are equal
and the distribution above the mean mirrors the distribution below the mean.
32 Because we are assuming a normal distribution, the percent ages (68, 95 and 99) will be evenly divided on either
side (above and below) of the mean.
-------
Page 50 Section IV: How to Get the Most out of Your Survey
D. Lessons Learned from Measurement Projects
One of the primary goals of this revised guide is to share lessons learned from the Regional pilot
projects and other sources. The lessons and experiences learned since the first version of the guide was
published in 1999 are reflected and incorporated throughout this document. This section lists some of the
notable lessons learned in the areas of general program evaluation, questionnaire design, and
questionnaire implementation, in order to give users a sense of the experiences of the Regions. Most of
these lessons relate to key messages of this guide: plan carefully, know the goals of your compliance
assistance and the purpose of your evaluation, use an appropriate collection tool, and know your target
audience.
General Program Evaluation
• Keep measuring reach and outputs. EPA emphasizes outcome measurement because
measuring for results provides the best indication of how well the Agency is protecting
human health and the environment. Outcome measurement, however, does not always
provide a complete picture of the quality of a particular activity. Continue to measure
reach to ensure that you are reaching your target audience. Also, continue to measure
outputs (e.g., the number of compliance assistance guides distributed, the number of
workshops conducted, the number of site visits conducts), because outputs show how
well the program is fulfilling its responsibility to reach out to its target audience. In
addition, where outcome data are unavailable or unclear, output data may be the best or
only source of performance information.
Questionnaire Design
Ask respondents to identify their position. It's not uncommon for individuals who are
not affiliated with a target facility, such as secondary compliance assistance providers, to
receive compliance assistance tools, attend compliance assistance presentations, or use
Web sites and helplines. Some of the pilot projects showed that, as respondents, these
individuals can skew results to make the compliance assistance appear less effective.
They automatically answer some questions in the negative because they are not in a
position to make a behavioral change or environmental improvement. If you cannot
eliminate these individuals from the respondent pool, consider asking them about the
nature of their position, so that you can sort your results appropriately.
Take language issues into account. Some of the pilot projects had a target audience that
included many non-English or English as a second language speakers. One pilot's
measurement efforts showed that it is important to have high quality translations of
compliance assistance and survey materials to avoid confusion and maximize
effectiveness. Translation services do require additional resources, however, and that
needs to be considered in your project planning as well.
-------
Section V: An Introduction to Statistical Sampling Page 51
• Match pre-/post-test questions to the compliance assistance material. If you know
you want to measure understanding through pre-/post-tests, develop your compliance
assistance material in advance to allow adequate time to put together matching test
questions. If your compliance assistance presentation, seminar, or workshop does not
follow a predetermined structure—for example, you are conducting a question and
answer workshop—or is formulated soon before it is delivered, you may not be able to
develop test questions that accurately assess the effectiveness of your efforts.
• Do not use comment cards. EPA and states found that these cards were potentially
biased and received very low response rates (usually around 2 percent). Such a low
response rate does not make the evaluation effort worthwhile and you should use another
data collection tool for your measurement projects.
Questionnaire Implementation
• Select an appropriate time-frame for conducting the evaluation. Some of the pilot
projects were affected by the high turnover rates in some industries. For one project,
waiting just six months between delivery and measurement of a compliance assistance
project resulted in a much lower response rate from employees leaving facilities. If you
are measuring changes in understanding, you might want to follow up sooner because of
the possibility of respondent turnover. At the same time, if you are measuring behavioral
changes or environmental and human health improvements, you will want to conduct the
evaluation after enough time has passed for changes to be implemented. If possible, try to
periodically follow up with recipients of compliance assistance.
• The Tailored Design Method works. Regional pilot results showed that the TDM can
produce significantly higher response rates for mailed surveys. Past pilots have also
shown that surveys that are not well-planned can have very disappointing results (past
OC response rates have been approximately 17%). The TDM suggestions in the guide
were selected because of their potential as resource-effective ways to get the most out of
your surveys.
• Direct observation works for measuring behavioral change. Several pilot projects
reported high satisfaction with the use of direct observation during on-site revisits. Direct
observation helped in making accurate assessments, measuring compliance changes,
obtaining detailed feedback from respondents, and observing innovative practices. Being
on site also helped compliance assistance providers learn about the language and culture
of their target audience and identify areas where future assistance might be provided.
• Get the most out of your pre-/post-tests. Several pilot projects showed that some
attendees will skip post-tests at the end of a workshop because they are tired or need to
depart for travel reasons. If possible, try to schedule pre-/post-tests immediately before or
after compliance assistance presentations when attendees are already assembled. You
should also let attendees know how long the tests are expected to take, so they do not
need to worry about committing too much of their time.
SECTION V: AN INTRODUCTION TO STATISTICAL SAMPLING
-------
Page 52 Section V: An Introduction to Statistical Sampling
This section provides basic information on how to draw a statistically valid random sample to
measure the performance of compliance assistance activities. After reviewing this section, you should be
familiar with a number of statistical concepts, be able to estimate the necessary sample size needed to
ensure statistical validity, and know how to draw the sample from your population. Please note that you
cannot use the generic ICR for a statistical evaluation; you will have to develop a separate ICR for any
statistical evaluations. (See Section III E for more information about obtaining OMB approval and using
or developing an ICR).
The information provided here is simplified and should not be considered a replacement for the
services of qualified survey statistician. This section is designed to assist you in developing a sampling
plan for a project that has few complications or special circumstances. Appendix F contains information
on advanced data analysis. For more complicated projects, you should consider using a qualified survey
statistician. Furthermore, you should consider having a statistician develop or review the sampling plan
for even simple projects where you will use the results of the survey to support important policy
decisions. Use the information in this chapter when:
You know the population from which the sample will be drawn; Appendix G contains
information to use when sampling from unknown populations.
You need to measure specific trends or aspects among the whole population (e.g., use of
compliance assistance tools among auto repair shops) or you need to divide the
population into subsets and look at trends within or among those subsets (e.g., use of
compliance assistance by independent auto repair shops and by shops that are part of
dealerships); and
Your main interest can be reduced to a "yes/no" binary-type question (e.g., in or out of
compliance, found the compliance assistance project useful or not).
This section identifies potential pit-falls, complications, and considerations that could warrant the use of a
qualified statistician.
This section presents definitions of some basic statistical terms, followed by a brief definition of
statistical validity. Next are aspects of a sampling plan that you will need to consider: the sampling frame,
the sample size estimate, and the sampling method.
-------
Section V: An Introduction to Statistical Sampling Page 53
A. Key Terms
Below are definitions of key statistical terms that you should know if you are going to conduct a
statistical sampling project. These terms will be used throughout this section.
Population—The complete group of entities that you wish to analyze. In terms of
compliance assistance programs, the scope of the program should define the population
for a sampling plan.
Sample—The subset of a population that participates in the survey. Taking a sample,
instead of measuring the whole population, reduces the cost of analyzing the whole
population. A measurement based on a sample is less accurate than if you measured the
whole population, but by adhering to statistical sampling principles you can obtain fairly
accurate estimates and, to some extent, control the resulting degree of inaccuracy.
Precision—The amount by which we are willing to be wrong in our measurement. A
good example of precision comes from political polling with statements such as "45
percent of people polled will vote for candidate X, with an error of 4 percentage points."
This means that 45 percent (plus or minus 4 percentage points) will vote for candidate x
(a range from 41 to 49 percent). The "4 percentage points" is the precision of the
estimated 45 percent. When you draw a random sample, you choose a level of precision
for your estimate, which is the amount by which you are willing to be wrong and still be
comfortable with your estimate.
Confidence Level— The probability that you are within your specified level of
precision. The actual percentages are based on statistical equations that won't be
discussed here. Confidence levels are generally set at 90 or 95 percent. Although 95
percent confidence levels increase the precision of your estimates relative to a 90 percent
level, where resources constrain the sample size that can be collected, EPA is comfortable
with use of a 90 percent level.
Unit of Response— The entity about which you are collecting information. For the most
part, a unit of response for compliance assistance performance measurement projects is
the facilities where the assistance is directed.
Bias—A sample is biased if the entities in the sample are not representative of the
population as a whole. For example, if a population of facilities that you are surveying is
evenly divided between small and large facilities and you collect data only from large
ones out of convenience, then the sample is biased. In collecting a sample, the idea is to
evaluate the population as a whole. To do this with some accuracy and confidence, you
must be assured that the sample is not biased in any way.
B. Statistical Validity
In drawing random samples to measure performance or compliance, you must follow standard
statistical techniques to ensure that the resulting measures and/or comparisons are statistically valid. In
-------
Page 54 Section V: An Introduction to Statistical Sampling
statistics, a valid study measures what you intend it to measure. Specifically, you can obtain a statistically
valid sample by using sampling procedures that result in a known precision and confidence for the
resulting measure and that obtain an unbiased estimate of the measure. Section D discusses how precision
and confidence requirements impact your choosing a sample size. In general, by adhering to statistical
sampling procedures, while also understanding the limits of these procedures, you should obtain
statistically valid measures.
C. Sampling Frame-The List of Potential Respondents
The sampling frame is the list of entities from which you draw your random sample. In its
simplest form, a sampling frame is a complete list of entities that you are interested in. Each entity on the
list will be "in scope" and no entities that might interest you are excluded. Developing an accurate and
complete sampling frame is one of the most important, and often overlooked, aspects of sampling.
The first step in developing a sampling frame is to define the scope of your effort. Usually, this
should match entities at which you have targeted the compliance assistance effort. You may be interested
in only the subset of the facilities that received compliance assistance (e.g., those that attended seminars,
to gauge the impact of the seminars). What is most important here is to develop a concise definition of the
facilities that are "in-scope." For each possible entity it should be clear whether or not it fits into the
scope.
In developing your sampling frame, you should understand the distinction between the target
population and the sampling population. The target population is the group of entities in which you are
interested and is defined by the scope of the effort discussed above. The sampling population is the group
of entities from which a sample is pulled. If a sampling frame is perfect, then the target population
matches the sampling population exactly. This is rarely the case, however. Developing a good sampling
frame involves minimizing the differences between the sampling population (defined by the frame) and
the target population (defined by the scope).
Differences between the target and sampling populations can result in serious biases that may
impact your results. Consider a survey that looks to measure compliance among dry cleaners (target
population) by drawing a random sample of dry cleaners that are participating in a voluntary program (the
sampling population). Unless all dry cleaners are part of the voluntary program, this survey will not be
able to make unbiased inferences about all dry cleaners. That is, we might expect that those who have
-------
Section V: An Introduction to Statistical Sampling Page 55
volunteered to be in the program will be different than those not in the program. If the sampling frame
and the target population do not match, then you will not be able to conduct unbiased analyses of your
target population. In actuality, in any sampling effort, you can only make unbiased inferences about your
sampling population since that is where you drew the sample from. Thus, it is important to have the target
population match the sampling population as best as possible.
EPA suggests that once you have a developed your sampling frame, take some time to determine
how well your list matches your target population. Here are some guidelines to develop and review your
sampling frame:
Coverage—How well does the sampling frame cover the target population? Does the
sampling frame include some entities that are not in the target population? If so, remove
them. Does the sampling frame omit any entities in the target population (e.g., a trade
association list may not cover all entities)? If so, can you add those entities?
Duplicates—Are there duplicates within the sampling frame? That is, does each entity
appear only once in the list? If entities appear more than once, remove the duplicates.
Current—How current is the list? Is the list several years old or was the list developed
recently? Obviously, the more current, the better.
Information Content— Does the sampling frame contain enough information for each
entity for the survey method that you will be using (e.g., telephone number, mailing
address, email address)? A frame that perfectly matches the target population, but is
missing addresses for several entities, will not support a mailed survey. Using such a
frame for a mailed survey and simply dropping those without addresses amounts to
changing, and potentially biasing, the sampling population to "those with addresses in the
sampling frame."
Stratification Criteria— If you are going to divide the sample into multiple groupings
(e.g., large, mid-sized, and small entities), you will need to ensure that your frame
contains the information to make those divisions. For example, you may want to divide a
set of auto repair shops between independent shops and dealer-based shops. You will
need to know into which category each shop falls. You should avoid assuming that
entities can be placed into certain groups without definitive information to back up that
assumption. This can be a serious issue where the frame is derived from more than one
source and the different sources contain varying levels of information.33
The sources for your sampling frame should be reliable and should provide all of the information
that you will need to determine which entities are "in scope" (e.g., entities that use a specific process) and
to administer your survey. For compliance assistance performance measurement projects, sampling
33 A related issue for stratifying multiple-source frames is that two sources may contain the same data element (e.g.,
employment levels), but one source's data may be more accurate or more reliable than the other sources being used.
-------
Page 56 Section V: An Introduction to Statistical Sampling
frames can come from sources such as:
Regulatory information;
Trade associations; and
Directories.
You should document the source for each entity in your final sampling frame. This information may assist
you in identifying biases after the survey has been completed.
You cannot always develop a sampling frame that closely matches your target population at the
beginning of the project. For example, suppose you are interested in determining compliance and the need
for compliance assistance among metal finishing facilities performing chrome electroplating. If there are
no reliable publications that document which facilities perform this operation, then developing an
accurate sampling frame at the start would be impossible. You can, however, use a sampling frame that
includes all metal finishers and then incorporate a screening stage into the sampling plan. In the screening
phase, you would contact entities to determine if they are "in scope" before they are asked to be a
respondent. This type of situation, however, is best handled by an experienced survey statistician.
An even more complicated situation would be where there is no reliable list of potential
respondents (i.e., the population is "unknown"). For these cases, we have provided some guidance on
sampling from unknown population in Appendix G. There are statistical sampling methods that can be
used to develop sample frames (e.g., cluster sampling), but implementation of these methods should be
left to an experienced survey statistician.
Finally, you should also consider if you may want to stratify (i.e., divide) the population.
Stratification can be done for two reasons. First, you may want to make valid statements about some sub-
groups of the population (e.g., a statistically valid compliance rate for small facilities). This is discussed
in more detail in Section D below. Second, you may simply want to ensure that some facilities of a certain
type are included in the sample (e.g., ensuring that the sample includes some small and some large
facilities when large facilities represent a small proportion of the population). Finally, you should also
consider how you may want to stratify (i.e., divide) the population. Specifically, your sampling frame will
need to contain the necessary information to stratify the sample. For example, if you want to stratify by
size (e.g., small versus large), you will need information on the size of each facility in your sample. This
can be a serious issue when you have used more than one source to develop your sampling frame. You
need to make sure that each source contains the necessary information. Additionally, you also need to
-------
Section V: An Introduction to Statistical Sampling Page 57
make sure that the information contained in each source is consistent. For example, a source that provides
specific numbers of employment size (e.g., 5,115) is probably not consistent with a source that provides
rounded numbers (e.g., 5,000)34. This has to be judged in context, however. In the above example, you
should review the number of facilities that may be close to the cut-off to determine the likely "error"
associated with inconsistent sources. A bigger issue would be where one source provides the necessary
information and another does not. In this situation, stratification may not be possible.
D. Estimating A Sample Size
You can estimate the sample size for a simple survey by answering six questions and then
applying one of two formulas. This section discusses those six basic questions and provides the two basic
formulas along with information on when each is applied. The method for estimating a sample size
presented here assumes that your survey is simple and straightforward. The discussion that follows points
out situations in which you may want to turn to a qualified survey statistician. Furthermore, if your
answers to these six basic questions do not fit neatly into the possibilities that this section provides, or if
you are unsure about your answers, you may consider using a qualified survey statistician.
Question 1: Are you measuring a proportion or an average value?
This question involves the nature of what you are measuring. To answer this question, think about
the factor that you are most interested in. Is it a compliance rate? Is it the use of a specific chemical?
Naturally, you may want to know several things about the entities, but what is the most important piece of
information? Ideally, this will be tied directly to your compliance assistance program. Once you have
determined that, can it be described as a proportion (e.g., the percentage of entities in compliance, the
percentage of entities using a certain chemical) or an average value (e.g., the average amount of a
chemical used)? In general, a proportion will be a yes/no question for the entities in the survey while an
average will be a measure that can be one of many values or a range of values for each entity. For simple
surveys, proportions are much easier to handle. For that reason, the remainder of this section will only
deal with sample sizes that are based on measuring proportions. Estimating a sample size for average
34 Rounding the specific source to match the rounded one may not solve the inconsistency problem. The problem
may stem from using different original sources for the employment numbers.
-------
Page 58 Section V: An Introduction to Statistical Sampling
values involves some additional complications that may be best handled by a survey statistician.35
Question 2: Are you concerned with measuring whether you are above (or below) a specific value
for your factor, or are you concerned with simply measuring the value of the factor?
How to answer this question is best illustrated by example. Suppose you have some information
(e.g., self-reported data) on the compliance rate in a sector and that information suggests compliance is at
90 percent. You may want to verify compliance is actually that "high" using a statistical sample. In this
case, you are concerned with measuring whether or not the factor (compliance rate) is below a specific
value. Suppose, on the other hand, you had no information or you were not concerned with simply
verifying a number. Instead, your goal is to simply "measure the compliance rate." In this case, you are
measuring the value of the factor. In the first case you are performing a "one-tailed test" (i.e., you are
concerned with only one side of a data distribution) while in the second you are performing a "two tailed
test". To determine which type of test is appropriate you should answer the two following sub-questions.
First, do you have some information on the value of the factor you are measuring? Second, do you wish
to verify whether or not the value is too high or too low? If you answer "yes" to both of these, then you
are conducting a one-tailed test. If you answer "no" to one or both of these questions, your situation is
best characterized by a two-tailed test. Most sample sizes can be drawn using two-tailed tests, which offer
more conservative sample size estimates. In other words, you can use a sample drawn using a two-tailed
test to answer one-tailed questions, but you cannot do the reverse and remain statistically valid. If you are
unsure about which type of test you are using, your best course of action is to assume a two-tailed test.
Question 3: With what level of error are you comfortable?
If you took a census of a population, you could be 100 percent certain of your results. In taking a
sample, however, you are making an estimate of some parameter that you then extrapolate, with some
uncertainty, to the total population. The purpose of sampling is to reduce the cost of measuring the
parameter, while obtaining an estimate that is close to the population value. Thus, your estimate will
involve some level of error. In using standard sampling procedures, however, you choose a level or error
with which you are comfortable.
35 Most notably, you must develop an estimate of the variance of the sample before you can estimate sample size,
which can be somewhat complicated. Sample sizes for proportions also require a variance estimate, but since
proportions are bounded by 0 and 1, the variance is also bounded. Thus, as a worst-case scenario, you could use the
maximum value for variance in drawing a sample for proportions.
-------
Section V: An Introduction to Statistical Sampling Page 59
In sampling for a proportion, you are actually choosing a range interval around your final
estimate that you are fairly certain contains the population value. As an example, suppose you chose an
error of five percentage points and your resulting estimate of a compliance rate was 65 percent. This
would mean that you are fairly certain the compliance rate for the population is between 60 and 70
percent (i.e., plus or minus five percentage points). One way to think of 'error' is that it defines the
interval that you are comfortable saying the population parameter falls in. In the example, the interval is
10 percentage points (two times the error we chose because the error applies to both sides of the estimated
value). If you had chosen a 10 percentage point error and obtained the same 65 percent compliance rate,
then your extrapolation would be much less precise: (i.e., you are fairly certain that population
compliance is between 55 percent and 75 percent). The more precise your estimate (smaller error), the
larger the sample size. In fact, you may want to calculate sample sizes for more than one error level.
The selection of error level will depend on what you consider to be relevant or acceptable.
Consider the above example where we used a 10 percentage point error. We are fairly certain that
compliance is between 55 and 75 percent. This level of precision may be fine for some projects, but too
imprecise for other. Additionally, if you are making a statistically valid comparison between two groups
(e.g., those receiving compliance assistance and those who do not) (see Question 6 below for more
details), an error that is too large may result in not detecting a difference in compliance rates. For any
project, you should determine which level or precision (or error) is acceptable to you given the goals of
the project.
Question 4: How confident do you want to be in your estimate?
Question 3 used the phrase "fairly certain" to describe the confidence that the population
compliance rate was within the interval. In estimating a sample size, you must also choose a level of
confidence, defined in probability terms. Accepted research methods, however, provide very clear
guidance on this aspect of choosing a sample size. In general, you should choose a level of confidence of
90 percent or higher; in reality, you need to consider only two levels of confidence: 90 and 95 percent.36
If you had chosen 95 percent in the above compliance rate example (with the 5 percentage point error and
65 percent estimate), then you are 95 percent confident that the population compliance rate is between 60
and 70 percent. Note, however, that there is still a 5 percent chance that the population rate is not within
that interval. Additionally, if you use a sampling frame that biases your results (see discussion above) or
36 You cannot choose 100 percent since you can only be 100 percent certain about your estimate if you perform a
census. Mathematically, 100 percent confidence reduces the formula for sample size to an undefined answer.
-------
Page 60 Section V: An Introduction to Statistical Sampling
collect data in a manner that biases your results (see discussion below), then these statements on
confidence and precision are meaningless. A sample size is only one aspect of choosing a valid random
sample. You must perform all of the other aspects correctly to ensure that your statements on precision
and confidence are valid.
Question 5: Are you concerned with making statistically valid statements about sub-groups of the
population?
This is an important question and can have significant implications for your final sample size, and
thus on the resources needed to administer your survey. As an example of this, you may be concerned
with measuring compliance among auto repair shops, and you have divided them into two groups:
independent shops and dealership-based ones. In this situation, you could (a) take a statistically valid
measurement for auto repair shops as a whole or (b) take a statistically valid measurement for each
subgroup individually. The second would imply the first, but requires significantly more resources. You
may want to do the second if your decision to implement a program to either subset depended on the
compliance rate for that group.
What often limits the use of taking valid measurement for sub-groups of a population is resource
constraints. In fact, you may decide that making statistically valid statements about subgroups is not a
priority after estimating a sample size for such a task. Making statistically valid statements about
subgroups of the population amounts to making statements such as the ones above (e.g., 95 percent
confident of the population value being within 5 percentage points on either side of the estimated sample
value) for each subgroup. To do this, you would need to treat each subgroup separately and estimate a
valid random sample size for each group. This could substantially increase the total sample needed for the
survey project. You could still estimate parameters for subgroups of the population without drawing a
valid random sample for those groups, but this will generally lead to weaker statements than if you had
drawn a valid random sample.37 Note that estimating valid parameters for subgroups of the population is
different than comparing two (or more) sub-groups to one another. This is discussed below in more detail.
37 From a statistical standpoint, you would leave the confidence level as is and let the error level (confidence
interval) increase. Thus, your resulting estimate would be less precise.
-------
Section V: An Introduction to Statistical Sampling Page 61
Question 6: Is the ultimate purpose of your sampling to compare two (or more) groups to one
another or to compare a single group to itself at different periods of time?
In many cases, you want to compare two samples. The most common situation in a compliance
assistance framework would be to compare a characteristic (e.g., compliance rate) before and after a
compliance assistance program is implemented. Another example would be where you want to compare
two different groups. For example, you may want to compare compliance rates among those that have
received compliance assistance versus those that have not. To make the comparison statistically valid,
you cannot simply draw two valid random samples (one before and one after). You must incorporate the
actual hypothesis test into your sample size calculation.38 As shown below, the formula for a two-sample
test differs substantially from a one sample test. To estimate the sample size for a comparison, you need
to ask Question 2 with the following changes:
What difference in the parameter (e.g., compliance rates) do you want to detect? In terms of
compliance rates, a 2-point difference may not be significant from a policy perspective, but a
10-point difference may be significant. You will need to determine the number of units that
represents a significant difference in terms of your programs or policies.
You can compare two drawn samples without using the two-sample comparison approach. Such a
comparison, however, would not be statistically valid (see definition above) and would result in weaker
statements regarding the comparison. Nevertheless, such a comparison may still be acceptable depending
on the purpose of your evaluation.
If the primary goal of your performance measurement project is to make a valid comparison, then
you should follow the two-sample approach. One form of valid comparison that is often done in a
performance measurement context is the "before and after" comparison (e.g., comparing compliance in a
sector before a compliance assistance program to compliance after a compliance assistance program has
been set up.) To perform a statistically valid "before and after" analysis you must use the two-sample
approach. If a comparison is not the primary goal, then taking two valid random samples and comparing
them using a standard hypothesis test should suffice.
38 A hypothesis test is a statistical comparison of an actual value (e.g., the proportion of facilities that said "yes" to a
specific question) to a hypothesized value. The comparison takes into account that the actual value may have some
variability associated with it (i.e., variance). The purpose of such a comparison is to determine if the actual value is
significantly different than the hypothesized value, given the variability of the actual value. Sample size estimates
are based on constructing a hypothesis test. Most basic statistics text books will contain a brief discussion of the
connection between sample size estimates and hypothesis testing.
-------
Page 62
Section V: An Introduction to Statistical Sampling
Formulas for Sample Sizes
You can develop sample sizes for estimating a proportion by answering the six questions above
and applying those answers to the formulas in Figure 5-1. If you are making statistically valid
comparisons between two groups, or comparing one group to itself at different times, then you should use
the two-sample formulas (Question 6). Otherwise, you should apply the one sample formulas. You would
also apply the one-sample formula to each subgroup of a population if you wanted to make valid
statements about each subgroup (see Question 5).
Estimating a sample size is a two-step process. First, you calculate a standard formula for either
the one-sample or two-sample scenario. Then, you adjust that sample size estimate using the adjustment
formula given in Figure 5-1. The adjustment is necessary for purely statistical reasons, but from an
implementation standpoint results in reducing the sample necessary for making a valid measurement.39
EPA suggests that you estimate several sample sizes by varying confidence and precision (error).
This will allow you to look at the trade-off between resources (i.e., larger samples require more resources)
and the statistical properties of your results.
Figure 5-1. Formulas for Estimating Sample Sizes
Sampling
Procedure
One sample
Two sample
Initial Estimate of Sample Size
„ - Z2PV~P)
d*
Z2CPl(l-Pl) +/»2(1-/>2)]
f
Final (Adjusted)
Sample Size Estimate
n- "°
1+^
N
Definitions
n0
n
N
The initial (raw) estimate for the sample size. In cases where the population is large, you
can use this as the sample size estimate.
The final sample size estimate, adjusted for population size. For the two-sample
comparison, the estimate applies to both samples (i.e., you need to pull a random sample of
n for both samples).
The population size.
39 In statistical terms, the initial estimate assumes an infinite population. This is rarely the case in sampling projects,
so you need to adjust the sample size estimate to reflect that your population contains a finite number of entities.
-------
Section V: An Introduction to Statistical Sampling
Page 63
Definitions
P> Pi' P2
z
d
The proportions that you are estimating, in numerical form (i.e., 50 percent isp = 0.5). In
the two-sample case, the subscripts refer to the two different samples. To calculate sample
sizes, you must use assumed values for these. If you have some prior information on the
proportion, use that information. For example, if you have anecdotal evidence that
compliance is about 60 percent, then use p = 0.6. In most cases, you will have no prior
information, so you would use 0.5 as the default value for the proportions. Using the
default value also produces to the most conservative estimate and hence the largest sample
size.
A statistical parameter that corresponds to the confidence level. It is the standard normal
score for a given level of confidence. The following criteria should be used to choose a z-
score value (see Question 2 for discussion of two- and one-tailed tests):
Two-tailed test: If you have chosen 90 percent confidence, then use z = 1.645. If
you have chosen 95 percent confidence, then use z = 1.96.
One-tailed test: If you have chosen 90 percent confidence, then use z = 1.29. If
you have chosen 95 percent confidence, then use z = 1.645.
The error level, or difference between two samples, that you have chosen. As with the
proportions above, use the numeric forms of the proportions (e.g., 5 percentage points is
0.05).
E. Drawing the Sample From the Population
After developing the sample size, you need to draw the random sample from the population. This
section discusses your sampling approach and how you adjust for non-response in your sampling plan.
Sampling Approach
The sampling approach specifies how you choose entities to be in the sample. In other words, if
you have a population of 100 entities and have estimated a sample size of 20, which 20 entities do you
choose? Clearly, the 20 should be randomly chosen, but how do you ensure randomness? Also, what if
you want to be sure that some members of each important group are included in the sample? Can you do
that without sacrificing randomness?
In most cases, you can choose a simple random sample from your population. To do this, simply
assign each entity a random number and sort by that random number.40 Your sample will then be the first
n entities, where n is your sample size.
40 Most spreadsheets contain a random number generator. Once you have assigned a random number to each entity,
change the random number formula to a value since most spreadsheet random number generators recalculate each
time the spreadsheet recalculates.
-------
Page 64 Section V: An Introduction to Statistical Sampling
Not all situations are this simple, however. For example, you may want to ensure that some
members of specific subgroups are included in the sample, but you may not want to use a valid random
sample for each subgroup due to resource constraints. In this situation, you can use a proportional
sampling framework which is generally referred to as stratification. To do this, divide the population list
into the relevant subgroups and calculate the proportion that each represents. For example, we may have
100 entities, 75 use process A and 25 which use process B. Given the large number using A, you can
select a sample of 20 without any process B entities. To ensure some representation of B entities, you
could choose 75 percent of our sample from the A entities (i.e., 15 entities) and 25 percent from the B
entities (5 entities). The key is to choose the sample randomly from each group. Thus, you follow the
simple random sampling procedures described above for each group. Note, however, that you do not have
a valid random sample for A entities or B entities individually, but a valid random sample for the
population of A and B entities as a whole.
One final possibility is to use a cluster sampling procedure. Cluster sampling involves dividing
the entities of the sampling population into "clusters," randomly choosing those clusters as a first step in
the sample draw, and then randomly choosing the entities within the clusters. Cluster sampling is a
procedure that is best left to qualified statisticians, but understanding when it may be useful could be
helpful. Specifically, a cluster sample is useful when:
You cannot develop a sampling frame due to logistical or practical reasons. In this
situation, use cluster sampling to develop the frame by first choosing clusters and then
entities within the cluster. For example, if you were looking issues related construction
sites in a certain state, a comprehensive list of those sites may be difficult or cumbersome
to obtain. One method to locate those sites, and thus form a sample frame, would be to
first divide the state into its counties and then randomly select counties. After that, you
could obtain lists of construction permits from each county and then randomly pull sites
from those lists.
Performing the survey on a given random sample would be expensive due to the
geographic distribution of entities. For example, you may be conducting site visits that
require substantial travel. You could reduce your costs if one trip included 2 to 3 site
visits. In this situation, you could develop a cluster sample based on location to reduce
costs and preserve the randomness of the sample.
Nonresponse Issues
So far, the discussion has assumed that entities will respond to your survey requests. In reality,
this is rarely the case. Section IV of this guide discusses methods of reducing nonresponse. Although
those methods are effective, completely eliminating nonresponse is almost impossible. If you anticipate
-------
Section V: An Introduction to Statistical Sampling Page 65
some nonresponse in your survey, adjust your sample size accordingly. The simplest way to do this is to
inflate the sample size to cover any anticipated nonresponse. For example, if you anticipate that 50
percent of all potential entities will not respond, then double your sample size. You should also track
who the nonresponders are, so that you can analyze any trends among nonresponders.41 If you suspect that
nonresponse will be a serious problem for your survey, however, you may want to consider using a
qualified survey statistician.
41 For example, if you have size information on the population and most of your nonresponders were "small"
facilities, then your results may be biased since large facilities would be over-represented in the data.
-------
Page 66 Section VI: OECA's Experience with the Dillman Method
SECTION VI: OECA'S EXPERIENCE WITH THE DILLMAN METHOD
AND STATISTICALLY VALID SURVEYS
In 2001, EPA's Office of Compliance (OC) tested methods for collecting outcome data from
compliance assistance efforts. The study compared two survey methodologies: mailed surveys using the
Dillman "total design method" and surveys conducted as on-site observations. The purpose of this
analysis was to test the hypothesis that the results of both data collection methods would be the same. For
this effort, EPA analyzed sectors in different stages of compliance assistance activity: 1) a sector for
which EPA is beginning a compliance assistance effort (marinas); and 2) a sector for which OC, other
EPA offices, and states have conducted several compliance assistance activities (metal finishing). EPA
also used the survey results to evaluate the impact of current compliance assistance activities and to
determine sector specific compliance assistance issues. However, this section focuses on comparing the
two data collection approaches.
A. Background and Purpose of the Study
EPA studied metal finishers and marinas to compare data gathering approaches and to evaluate
the effectiveness of the Dillman method for collecting compliance assistance evaluation data. Due to
resource limitations, EPA focused the metal finishing study on EPA Regions 1 and 5. EPA focused the
marinas study on EPA Region 1 since this region was beginning a marinas compliance assistance
initiative. Table 6-1 summarizes background information for each study, including the target population,
the survey scope, consultations conducted for survey development, and the source of the list used to
develop each sample frame. EPA conducted the following steps for each sector:
Develop the survey instruments;
Determine the number of survey respondents required for each survey (see Section 6.B);
Develop the list of recipients for the mailed and on-site observations (see Section 6.B);
Use the Dillman method for the mailed survey (see Section 6.C);
Arrange and conduct the on-site observations (see Section 6.C); and
Analyze the results (see Section 6.D).
Although the metal finishing and marinas survey projects both consist of a mailed survey and on-
site observations, the results from the two sectors are not directly comparable. The goals of the two
projects were vastly different. The metal finishing survey project was designed to evaluate the relative
usefulness of mailed surveys and on-site observations. The metal finishing survey focused on awareness
of regulations and general indicators of environmental performance. The marinas survey project, on the
-------
Section VI: OECA's Experience with the Dillman Method
Page 67
other hand, was initially designed to collect baseline compliance information through on-site visits.
OECA added a mailed survey to this project to evaluate the relative effectiveness of each survey mode.
Given the differing original goals of the two projects, comparing the results is not valid. Additionally, the
two sectors are vastly different with respect to their experience with environmental regulations. The metal
finishing sector has a great deal of experience with environmental regulations, but, in contrast, the
marinas sector has very little regulatory experience. This difference may affect the level of regulatory
awareness that each sector has.
In developing the survey instruments, EPA consulted with a variety of industry representatives
and compliance assistance providers. Table 6-1 provides more information on the groups consulted for
this effort. For each sector, EPA used the same survey instrument for both the mailed survey and the on-
site observation (i.e., for the metal finishing study, the same survey instrument was used for both the
mailed and on-site observations). Figure 6-1 presents the metal finishing survey. Figure 6-2 presents the
marina survey. These surveys are included at the end of the section.
Table 6-1. Background Information on 2001 EPA Data Collection Study
Metal Finishing Sector
Marinas Sector
Goals
The goals for the metal finishing sector
survey were to (1) collect information
through two survey modes and compare the
costs and results of the two survey modes
and (2) collect basic information on the
metal finishing sector that could be used in
developing compliance assistance programs.
The goal of the marinas survey was to collect
baseline compliance information using both self-
reported information and an on-site visit
program as part of an effort to measure results of
Region 1's environmental assistance program.
Self-reported data were collected through a
mailed survey to compare to the on-site data
collection.
Target
Population
Metal Finishers in EPA Regions 1 and 5.
These regions have conducted extensive
compliance assistance for metal finishers.
Marinas in EPA Region 1. This region was
initiating a compliance assistance program for
marinas.
Survey
Instrument
Consultation
In developing the metal finishing survey
instrument, EPA/OC consulted with:
• Members of the metal finishing industry;
• Metal finishing trade associations;
• State compliance assistance staff; and
• Other EPA offices.
In developing the marinas survey instrument,
EPA Region 1 conducted stakeholder meetings
to consult with:
1. Marina owners;
2. Four state marina trade association
presidents;
3. State compliance assistance providers;
4. State Coastal Zone Management program
offices; and
5. Small Business Development Center staff
members.
-------
Page 68
Section VI: OECA's Experience with the Dillman Method
Metal Finishing Sector
Marinas Sector
Sampling
List Data
Source(s)
The Harris Directory - A database with
information on more than 700,000 U.S.
businesses, covering the manufacturing
sector. Information includes company name,
location, size, and industry type (determined
by SIC code). EPA purchased a list of all
facilities in SIC codes 3471 and 3479 in the
EPA Region 1 and 5 states.
State Coastal Zone Management program
offices provided much of the marina list.
Additional information for inland sites was
obtained from tourism and boating guides and
state boating maps.
B.
Methodology
Estimating Sample Size
To estimate a sample size for this project, EPA followed the methods discussed in Section V of
this guide. Table 6-2 provides EPA's answers to the six questions posed in Section V. Based on the
answers to the six questions, EPA applied the sample size formula for a two-tailed, one-sample procedure
found in Figure 5-1 along with the suggested adjustment to the initial estimate. EPA calculated that a
sample size of 92 to 94 facilities was necessary for the metal finishing sector and that a sample size of 70
facilities was necessary for the marinas sector.42'43
Table 6-2. Factors Contributing to Sample Size Estimate
Question
Metal Finishing and Marinas Survey
1: Proportion or Average Value?
EPA's goal (for metal finishing) was to measure the percentage of
facilities that had received compliance assistance. This implies that the
formulas for proportional sampling (found in Section V) can be used.
Additionally, EPA had no information on this proportion to construct an
initial estimate. Therefore, for purposes of estimating the sample size,
EPA assumed/) = 0.5 in the Figure 5-1 equations.
42 The ranges stem from the use of ranges for the population size in the adjustment formula of Figure 5-1. For metal
finishing, a population range of 2,000 and 4,000 facilities was used. For Marinas, a population of 1,200 facilities was
used.
43 During the sample facility selection phase for marinas, however, one state exited from the study. As discussed
later in this section, both the metal finishing and the marinas studies involved stratifying the sample size by state. To
account for the exited state, the sample size for marinas was reduced by 30 facilities which was the number that the
state accounted for in the stratification process.
-------
Section VI: OECA's Experience with the Dillman Method
Page 69
Question
2: One- or Two-Tailed?
3 : Level or error (precision)?
4: Confidence?
5: Statistically valid stratification?
6: Statistically valid comparisons?
Metal Finishing and Marinas Survey
EPA designed this analysis as a two-tailed test to learn the compliance
rate rather than the percentage above an assumed rate. This implies using
the z values for a two-tailed test found in Figure 5-1 (see question 4
below for an exact value).
EPA chose a 10 percentage point level of precision, or d = 0. 10 in the
Figure 5-1 formulas.
EPA chose a 95 percent level of confidence. Combined with question #2,
this implies using a z value of 1.96 in the Figure 5-1 formulas.
EPA did not stratify the sample for purposes of making statistically valid
statements about subgroups of the population. EPA did, however, stratify
to ensure representation of the different sizes, states, and types of
facilities. Therefore, the sample will represent the population as a whole
and not a subgroup.
EPA's goal was not to make statistically valid comparisons. This implies
using the one-sample formula.
Stratification
Metal Finishing
EPA stratified the random sample by both state and size (defined by number of employees) for
both the mail survey and the on-site observation. This was done to ensure that each of the states in EPA
Region 1 (CT, ME, MA, NH, RI, VT) and EPA Region 5 (IL, IN, MI, MN, OH, and WI) were
represented in the surveys and that both small and large facilities were included in the sample.44 For this
project, EPA defined any metal finisher with less than 10 employees as a "small" facility and all other
facilities as "large."
EPA stratified first by state by using information contained in the sampling frame to determine
the percentages that each state represented in the population. EPA applied these percentages to the sample
size estimate to determine the number in the sample from each state. Next, EPA used information from
the sampling frame to determine the percentage of the population within each state that employed less
than 10 employees. EPA applied these percentages to the sample size for each state to determine the
number of small and large facilities to draw from each state.
44 Although EPA intended to ensure that all states and sizes were represented, EPA did not intend to make
statistically valid measurements of these subgroups (i.e., each state and large and small facilities).
-------
Page 70 Section VI: OECA's Experience with the Dillman Method
Marinas
EPA stratified the sample for marinas by both state and location (inland versus coastal) for both
the mailed and the on-site observations. As with the metal finishing surveys, the stratification was done to
ensure that each state and both location types were adequately included in the sample taken.
EPA stratified first by state by using information contained in the sampling frame to determine
the percentages that each state represented in the population. EPA applied these percentages to the sample
size estimate to determine the number in the sample from each state. EPA then used the percentage in
each state that were inland and costal marinas in the sampling frame to estimate the number of each
marina type in each state's sample.
Sample Selection and Nonresponse Replacement
Metal Finishing
EPA conducted the surveys on this sector as a blind voluntary random sample (i.e., the facility
identity is unknown to the Agency and facilities participated voluntarily). To draw the random sample,
EPA first divided the population into separate lists for each state and then within each state by size class
(i.e., those with less than 10 employees and those with 10 or more employees). EPA assigned each facility
a random number (using Microsoft Excel's random number generator) and then sorted each list based on
random number.
For the metal finishing sector, EPA anticipated at least a 50% response rate on the mailed surveys
using a four step contact process. To account for this, EPA doubled the sample size estimate for mailed
surveys. As a result, EPA actually mailed 202 surveys to facilities, more than twice the initial sample size
estimate.
EPA anticipated that each facility visited in the site observation would result in a completed
survey. In the event that some facilities did not agree to participate in the site visit program, EPA
continued to pull facilities from the sorted list of facilities. A number of facilities refused to participate
(see Section 6D, Results below). Project schedules forced EPA to discontinue the site visit phase after
visiting 67 in-scope facilities.
-------
Section VI: OECA's Experience with the Dillman Method Page 71
To ensure that no facility appeared in both the mailed and visited samples, EPA drew both
samples simultaneously. The first 2x facilities in each sorted list of state/size combinations constituted the
random sample for the mailed survey, where x was the original sample size estimate for each state and
size combination. The next x facilities were the initial set of contacts for the site visit observation. If a
member of the site visit group refused to participate or was found to be out of scope, EPA contacted the
next facility on the list. In total, EPA drew a random sample of more than three times the desired range,
randomly assigning two-thirds of the sample to the mail survey and one-third to the site visits.
Marinas
EPA conducted the marinas survey as a voluntary random sample. EPA selected the marinas for
the on-site observations first, and then set up the required number of visits. Facilities that were not
scheduled for an on-site observation were then used as the frame for the mailed survey sample. Thus, it
was not possible for any facility to be in both samples.
EPA used a random selection process to develop the list of marinas to contact for the on-site
observations. Specifically, EPA developed a final list for each state and then chose every tenth marina
from the list until the required number of marinas was drawn for each state. EPA then reviewed the list to
determine how many were inland versus coastal. If the required number of inland marinas was not
attained for each state, EPA replaced the final coastal ones that were drawn with inland marinas that were
also drawn randomly.
As with the metal finishing survey, some facilities declined on-site observation requests by EPA
or were not in scope for the survey effort. To replace these facilities, EPA located the facility that had
declined or was not in scope in the sample frame list and then selected the next facility in the list. If that
replacement facility declined or was not in scope, EPA selected the one directly before the original
selected facility. Additional replacement facilities were chosen in this manner, moving progressively
further from the original selected facility and alternating between succeeding and preceding facilities.
EPA anticipated at least a 50 percent nonresponse rate for the mailed survey and in response more
than doubled the number of surveys mailed. In total, EPA mailed 236 surveys to marinas. To choose the
random sample for marinas, EPA followed an identical process as that for the metal finishing mailed
survey.
-------
Page 72 Section VI: OECA's Experience with the Dillman Method
C. Implementation
Mailed Surveys
In accordance with the Dillman method, EPA sent each facility a prenotification letter. This letter
was signed by the appropriate regional authority and copied onto EPA regional letterhead. The letter
introduced the project, described the purpose of the study, notified the facility of the imminent arrival of a
mailed survey, and informed them that participation was voluntary. Appendix E contains a copy of the
metal finishing prenotification letters.
The survey and a signed, copied, cover letter followed approximately two weeks later. The cover
letter reiterated much of the prenotification information. It also provided a toll-free number to call with
any questions regarding the survey or the program itself. Although the cover letter was copied onto EPA
letterhead, the survey was mailed in a 9x12 manilla envelope showing the EPA contractor's return
address information, rather than EPA's. This was done to emphasize the fact that this was a blind (to
EPA) study. Each survey was printed with a randomly chosen site identification number to help the
contractor evaluate if the sample frames (i.e., states, sizes) were equally represented in the responses. A
stamped self-addressed return enveloped was also included with the survey to help increase the return
rates. Appendix E contains a copy of the metal finishing cover letter.
Once the facilities had been given adequate time to complete and return the surveys (2-3 weeks) a
reminder postcard was sent to all facilities that had not yet responded. Each postcard was oversized (8.5"
x 5") and printed on blue cardstock to make it more visible to the recipient. The post card text reminded
facilities of the project and survey, thanked them if they had already participated, and set a deadline for
returning the survey. Appendix E contains a copy of the metal finishing reminder postcard.
Finally, facilities that did not return the survey two weeks after receiving the postcard were sent a
second survey. The re-mail package contained a modified cover letter. In addition to all of the previous
information, this letter stated that the facility had already received a survey but had not yet participated.
This package also contained a stamped, self-addressed return envelope. Appendix E contains a copy of
the metal finishing re-mail cover letter.
As facilities returned their surveys, EPA's contractor logged them into a database. If any of the
materials were returned as undeliverable, the facility was removed from the remaining mailing lists.
-------
Section VI: OECA's Experience with the Dillman Method Page 73
These facilities were not considered as part of the "returned" surveys discussed in the "Results" section.
Site Visit Observations
As stated above for both the metal finishing and the marinas on-site observations, EPA selected a
random set of facilities to contact for site visits. For the metal finishing sector, these facilities received a
prenotification letter very similar to that of the survey recipients but this one notified of an upcoming
phone call to schedule a site visit. Within two weeks the facilities were called to schedule an appointment.
If a facility chose not to participate, could not be reached, or was not currently performing metal finishing
activities, then a replacement facility was taken from the list. Replacement facilities, which were
randomly selected at the beginning of the process, were chosen based on both size and state. For example,
if a small site in Connecticut chose not to participate, then the next small, Connecticut site was taken from
the top of the replacement list. This process was repeated until either the total desired number of site visits
were scheduled or the project schedule/budget was expended.
For the marinas sector, once a set of facilities was selected, each facility was contacted to
determine eligibility (i.e., in scope or not) and willingness to be involved in an on-site observation.
Facilities that were either out-of-scope or declined a visit were replaced using the process described in the
previous section. To encourage participation, the marina owner was told that assistance would be offered
during the visit and it was clarified that the visit was not an official enforcement inspection.
D. Results
This section compares responses to the two survey modes (e.g., response rates). We compare the
two samples with respect to their general characteristics, the types of compliance assistance received, and
the understanding of regulatory requirements. These three comparisons support the inference that the
samples are similar in nature (i.e., the respondents for each survey mode had similar characteristics).
Finally, we compare the performance towards key environmental regulations.
Response Comparison
-------
Page 74 Section VI: OECA's Experience with the Dillman Method
Several aspects of response rates are relevant to compare for the two samples. We begin by
discussing how we calculated the overall response rate for each survey mode. Next, we focus on the
details of response for the mailed surveys and then the details of response for the site visit observations.
Finally, we compare the resource requirements to implement each survey mode on a per response basis.
Overall Response Rates
To calculate response rates, EPA followed guidance from both the Council of Survey Research
Organizations (CASRO) and Lohr (1999).45 Both sources suggest adjusting for out-of-scope entities
among unreturned surveys in calculating response rates. In cases where we can be sure that all members
of the sampling frame are in-scope, dividing the number of returned surveys by the total number of
surveys sent will provide an accurate response rate. In most survey situations, including this one, we
cannot be sure that all members of the sampling frame are also in-scope. Thus, some entities may not
return a survey since they are out of scope. Survey researchers tend to agree that out-of-scope facilities
that do not respond should not be counted as non-re spenders when calculating response rates. Therefore,
EPA adjusted for this by estimating the number of out-of-scope facilities that did not respond.46
45 The CASRO guidance was the result of a task force developed by CASRO to provide definitive guidance to
survey researchers on how to calculate response rates.
46 In the formulas that follow, this is done in the denominator of each response rate. The first term of each
denominator multiplies the percentage that are in-scope among the returned surveys (e.g., CIS/TR in the mailed
survey response rate) by the number of unreturned surveys (NR). Thus, the first term provides an estimate of the
total in-scope non-responders by using information from the returned surveys.
-------
Section VI: OECA's Experience with the Dillman Method
Page 75
We used the following formulas to calculate response rates (RR):
Mailed Surveys
RR =
CIS
CIS
TR
*NR + CIS
RR = Response Rate
CIS = Completed in-scope surveys
TR = Total Returned
NR = Number Not Returned
Site Visits
RR = —
IS
IS + OS
+ IS
IS = In-scope facilities = V + M + R
V = Number of visits.
M = Facilities that agreed to receive visit, but we were
unable to visit.
R = In-scope facilities that refused visits.
U = Unknown (unreachable)
OS = Out-of-scope facilities.
In developing these response rate formulas, EPA concentrated on developing estimates that could
be compared across the two survey modes. Table 6-3 presents the estimated response rates for both
sectors and both survey modes.
Table 6-3. Response Rates
Mode
Mailed
Site Visits
Metal Finishing
43.6%
40.7%
Marinas
51.3%
94.6%
For the metal finishing sector, both evaluation modes resulted in response rates less than 50
percent. Additionally, the mailed survey resulted in a slightly higher response rate (by 2.9 percentage
points) than the site visits. Normally one would expect a significantly higher response rate for the site
visits; however, there is a high level of distrust towards the regulatory agencies and therefore a degree of
hesitancy towards voluntarily allowing them on site. In addition, the mailed survey response rate is much
higher than the 17 percent that OC has experienced in the past. This success is mostly attributed to the
Dillman method. The marinas' surveys both had response rates over 50 percent, albeit the mailed
response rate was only 51.3 percent. The site visit response rate, however, was close to 95 percent,
indicating a high degree of success in getting facilities to agree to participate. These differences between
the programs may be indicative of a sector in the midst of a contentious regulatory development program
(metal finishing) and one that has no negative history with EPA (marinas).
-------
Page 76
Section VI: OECA's Experience with the Dillman Method
Details on the Response Process for Both Mailed Surveys and Site Visit Observations
Table 6-4 presents information on the return process for the mailed surveys for both the metal
finishing and the marinas sectors. As shown in Table 6-4, more than half of all responses resulted from
the initial mailing for both surveys. Additionally, the initial mailing and the postcard generated more than
70 percent of all responses in both sectors (80.9 percent for metal finishing and 71.8 percent for marinas).
Comparing the total mailed to total in-scope returns, we see that the metal finishing survey
received one in-scope response for every three mailed surveys, while the marinas sector received one in-
scope response for every two mailed surveys. This is likely due to the data sources used to identify
facilities in each sector. Since the metal finishing study included a broader geographic range than the
marinas sector, EPA used a national database (the Harris Directory) to identify recipients. This database
likely included more out-of-scope facilities than the more local databases used for the marinas survey.
Additionally, more refinement and cross-checking was done for the marinas sampling frame due to the
lack of one comprehensive source for a sampling frame as in the metal finishing surveys.
Table 6-4. Mailed Survey Returns
Survey Status
Returned After Initial Mailing
Returned After Reminder Postcard
Returned After Second Mailing
Total In-Scope Returns
Total Mailed
Metal Finishing
52
(76.5%) [a]
3
(4.4%) [a]
13
(19.1%) [a]
68
202
Marinas
71
(60.7%) [a]
13
(11.1%) [a]
33
(28.2%) [a]
117
236
[a] As a percentage of total in-scope returns.
Table 6-5 presents similar information for the site visits. For the metal finishing sector, a little
more than half of all site visits (53.7 percent) came from the original sample. Additionally, comparing the
total number of visits to the total number of contacts, it was necessary to contact 4.25 facilities to obtain
each site visit in the metal finishing sector. Details on originals versus replacements were not available for
the marinas on-site observations. Comparing total contacts to number of visits, however, the marinas
on-site observation project needed only 2.1 contacts to generate the required number of in-scope visits.
The discrepancy between the number of contacts per in-scope visits may be associated with degree of
-------
Section VI: OECA's Experience with the Dillman Method Page 77
prior EPA attention for each sector. That is, the marinas sector has received much less attention in the past
from EPA compared to the metal finishing sector.
Table 6-5. Distribution of Responses for Site Visits
Contact Stage/Status
Visits - Original Sample
Visited - Replacement Sample
Total Visits
Total Contacted Sites
Metal Finishing
36 (53.7%) [a]
31 (46.3%) [a]
67
285
Marinas
NA[b]
NA[b]
70
146
[a] As a percentage of total visits.
[b] This information was not available for the marinas on-site observations.
Budget-Cost Per Response
Table 6-6 presents information on the cost per in-scope response for both sectors and both survey
modes. Both the hours and the nonlabor costs included in Table 6-6 only reflect those items that are
different between the two survey modes (e.g., the hours required for developing the survey instrument
and the mailing list and the costs for the Harris Directory are not included since these would be required
for each survey mode). Although not included in the costs and hours below, EPA has found that defining
the target population, developing the survey instrument, and acquiring a useable recipient list can be the
most time consuming portions of the project.
When evaluating the numbers presented in Table 6-6, consider two important factors: 1) the
limits on the types of information that can be collected through a mailed survey, and 2) the value of
additional anecdotal information that can be collected through on-site observations. As discussed later in
this section, mailed surveys are useful for obtaining general information on awareness of environmental
regulations or information on trends in environmental performance. However, mailed surveys may not
yield accurate data on actual compliance rates; this type of information can be collected during on-site
observations. Also, during on-site observations, facility personnel may become comfortable enough to
openly discuss compliance concerns, root causes of non-compliance, and compliance assistance
preferences. This type of anecdotal information is helpful in designing compliance assistance programs.
For the metal finishing sector, each in-scope on-site observation required an additional 9.9 labor hours
and approximately $561 in costs. For the marinas, each in-scope on-site observation required an
additional 5.3 labor hours and approximately $274 in costs. Much of these differences can be accounted
-------
Page 78
Section VI: OECA's Experience with the Dillman Method
for by additional travel time and money required by the multi-regional scope of the metal finishing
project.
Table 6-6. Cost Per In-Scope Response
Category
Metal Finishing
Marinas
Site Visits
Labor Hours Per In-Scope Response
Labor Cost Per In-Scope Response
Nonlabor Costs Per In-Scope Response
Total Cost Per In-Scope Response
11.0 [a]
$435 [b]
$177 [c]
$612 [d]
6.4 [g]
$253 [b]
$72 [h]
$325 [d]
Mailed Surveys
Labor Hours Per In-Scope Response
Labor Cost Per In-Scope Response
Nonlabor Costs Per In-Scope Response
Total Cost Per In-Scope Response
1.1 [e]
$43 [b]
$8[f]
$51 [d]
[a] Hours include training, phone calls to set up visits, travel time, on-site interview time.
[b] Calculated by multiplying the hours by a loaded hourly rate of $3 9.51. The loaded hourly rate reflects use of a
GS-13, Step 1 employee (Washington, DC area pay scale) in 2001. The annual salary for that employee is $63,211,
resulting in a hourly rate of $30.39. We added a 30 percent mark-up to this hourly rate to get the loaded rate of
$39.51.
[c] Costs include only those costs unique to site visits, such as travel-related costs (airfare, hotel, car rental, food) for
site visits and long-distance phone charge charges.
[d] This is the sum of labor and non-labor costs.
[e] Hours include coordination with Regions, copying, filling envelopes, and logging survey returns.
[f] Costs include only those costs unique to mailed surveys, such as copying, postage, envelopes, labels, and printing
postcards.
[g] Hours include travel time, facility research, training, phone calls, and designing a visit protocol.
[h] Costs include travel-related costs associated with hotel, car rental, food, and incidentals.
Comparing Characteristics and Survey Responses
Metal Finishing
Tables 6-7 through 6-9 summarize characteristics of the two samples for the metal finishing
sector. Each characteristic was represented by a yes/no question in the survey. This information compares
the two samples and determines how similar they are to one another. If there are few differences in these
characteristics, then the two survey modes have essentially drawn comparable samples, and differences
may be attributable to the survey mode. A large number of significant differences would indicate that
-------
Section VI: OECA's Experience with the Dillman Method
Page 79
respondents to one survey mode are different than the respondents to the other mode.
To formally compare the two samples, EPA performed statistical hypothesis tests. These tests
looked for statistically significant differences between the two samples in terms of the percentage that
IP mipctinn. That is, did one sample have a significantly larger percentage that said "yes"
said "yes" to the question.
to
the question than the other sample. The results of these comparisons can be summarized as follows:
General Characteristics (Table 6-7)—For the most part, the two samples are identical
in terms of general characteristics. There are, however, two significant differences. A
larger proportion of the visited facilities generate both metal-bearing wastewater and
hazardous waste compared to the mail survey facilities. This is not surprising for the
following reasons. If a facility generates cyanide- or chromium-bearing wastewater, it
usually also generate metal-bearing wastewater. These facilities are typically aware of the
constituents in their wastewater and their regulatory requirements. However, if a facility
generates only metal-bearing wastewater (e.g., they may only be performing barrel
finishing or cleaning operations prior to dry surface finishing operations), it is possible
that they would not consider their wastewater "metal-bearing" when completing the
survey. The site-visit personnel, however, would classify this as metal-bearing
wastewater because of expected metals concentrations. With respect to the hazardous
waste generation, several site-visit personnel identified facilities that did not realize that
they were generating hazardous waste even though they were aware of hazardous waste
requirements; therefore, it is not surprising that the mailed surveys indicated fewer
hazardous waste generators than the site visits.
Compliance Assistance (Table 6-8)—The types of compliance assistance received by
both samples is generally similar. However, a larger proportion of visited facilities
acknowledged to receiving wastewater compliance assistance compared to the mail
survey facilities.
Awareness (Table 6-9)—There were no significant differences between the two samples
in terms of facilities' understanding of regulatory requirements.
Based on the few significant differences, the two metal finishing samples appear to be similar.
Table 6-7. General Characteristics of the Two Samples in the Metal Finishing Sector
Characteristic
Percent Answering "Yes"
Mailed Survey
(n = 68)
Site Visits
(n = 67)
Statistically
Significant
Difference
Wastewater
Generate Chromium-Bearing Wastewater
Generate Cyanide-Bearing Wastewater
Generate Metal-Bearing Wastewater
36.8%
23.5%
47.1%
37.3%
20.9%
64.2%
No
No
Yes
-------
Page 80
Section VI: OECA's Experience with the Dillman Method
Characteristic
Have a Discharge Permit
Percent Answering "Yes"
Mailed Survey
(n = 68)
60.3%
Site Visits
(n = 67)
56.7%
Statistically
Significant
Difference
No
Hazardous Waste
Generate Hazardous Waste
48.5%
62.7%
Yes
Air Emissions
Perform Chromium Electroplating
Have a Solvent Cleaning Machine
22.1%
19.1%
19.4%
19.4%
No
No
Toxic Release Inventory
TRI Exempt
29.4%
34.3%
No
Table 6-8. Types of Compliance Assistance Received For the Two Samples
in the Metal Finishing Sector
Compliance Assistance Received
Wastewater
Hazardous Waste
Air Emissions-Chromium Electroplating
Air Emissions-Solvent Cleaning Machine
TRI Reporting
National Metal Refinishing Resource Center web site
Percent Answering "Yes"
Mailed Survey
17.6%
(n=43)
22.1%
(n=33)
8.8%
(n=15)
5.9%
(n=13)
20.6%
(n = 68)
13.2%
(n = 68)
Site Visits
32.8%
(n=47)
32.8%
(n=42)
6.0%
(n=13)
7.5%
(n=13)
25.4%
(n=67)
9.0%
(n=67)
Statistically
Significant
Difference
Yes
No
No
No
No
No
Table 6-9. Awareness of Regulatory Requirements for the Two Samples
in the Metal Finishing Sector
Awareness
Hazardous Waste Labeling Requirements
Percent Answering "Yes"
Mailed Survey
97.0%
(n=33)
Site Visits
95.2%
(n=42)
Statistically
Significant
Difference
No
-------
Section VI: OECA's Experience with the Dillman Method
Page 81
Awareness
Hazardous Waste Manifest Records Requirements
Air Emission Control Requirements for Chromium
Electroplating Tanks
Air Emission Standards for Solvent Cleaning Machines
TRI Form R
National Metal Finishing Strategic Goals Program
Percent Answering "Yes"
Mailed Survey
97.0%
(n=33)
100.0%
(11=15)
61.5%
(n=13)
61.8%
(n = 68)
23.5%
(n = 68)
Site Visits
95.2%
(n=42)
100.0%
(n=13)
69.2%
(n=13)
53.7%
(n=67)
26.9%
(n=67)
Statistically
Significant
Difference
No
No
No
No
No
Marinas
Table 6-10 summarizes the characteristics of the marinas in both the mailed and on-site
observations. As with the information for the metal finishing surveys, EPA performed statistical tests to
determine if significant differences existed between the samples in each mode. Unlike the metal finishing
survey, however, a number of the questions we assess in Table 6-10 are quantitative in nature rather than
yes/no questions. For the quantitative questions, we compare the average value between the two surveys.
We compared eight characteristics between the two samples and in four cases there was a
significant difference. Most notably, the facilities contacted through the mailed surveys had more full-
time employees than the on-site observations, but the on-site observation facilities had a larger number of
boats (total boats moored, docked, and stored on land) than the mailed survey marinas. Additionally, of
those marinas where maintenance and repair operations take place, a significantly larger proportion of on-
site observations had primarily customer do-it-yourself maintenance and repair operations compared to
the mailed survey sites. These three significant differences (number of full-time employees, total number
of boats, and primarily customer do-it-yourself maintenance and repair) make it difficult to conclude that
the two samples are similar.
Table 6-10. Characteristics of Marinas in Both Samples
-------
Page 82
Section VI: OECA's Experience with the Dillman Method
Characteristic
Mailed Survey
On-Site
Observations
Statistically
Significant
Difference
Number of Employees
Average number of full-time employees
Average number of part-time employees
13.6
(n=117)
3.7
(n=117)
8.8
(n=70)
4.3
(n=70)
Yes
No
Number of Boats
Average number of moored boats
Average number of docked boats
Average number of boats stored on land
Average number of boats that are moored, docked, and
stored on land at the facility
31.3
(n=117)
70.7
(n=117)
73.8
(n=117)
175.9
(n=117)
27.1
(n=70)
96.5
(n=70)
94.9
(n=70)
218.5
(n=70)
No
Yes
No
Yes
Maintenance and Repair Operations
Percentage that indicated maintenance and repair
(M/R) operations are performed at the marina
Of those that indicated M/R operations are performed,
the percentage that indicated the M/R operations are
primarily customer "do-it-yourself ' operations.
90.4%
(n= 114)
16.2%
(n= 105)
94.2%
(n = 69)
40.6%
(n = 69)
No
Yes
Comparison of Performance Towards Key Environmental Regulations
Metal Finishing
Table 6-11 compares the results of the on-site observations to the mailed surveys for a number of
compliance-related questions. The purpose is to compare the environmental performance reported in the
mailed surveys to that observed on-site, given the assumption that the on-site observations would generate
accurate information. We can then evaluate the correlation between the mailed survey and|observed
information to assess if the mailed survey respondents biased their responses towards better
environmental performance.
In total, we compared answers to 15 questions that specifically requested information on
environmental performance. Table 6-12 summarizes the numbers and percentages of questions where
mailed respondents had (a) better reported performance, (b) statistically significant better reported
-------
Section VI: OECA's Experience with the Dillman Method Page 83
performance, and (c) a reported performance five percentage points or more higher than on-site
observations (considered to be numerically significant). For eight of the 15 questions, the mailed
re spenders indicated better performance towards key environmental regulations. On the remaining seven,
the performance was either the same between the two modes or the on-site responses had better
performance. For two questions, the compliance rate for mailed respondents was larger by a statistically
significant margin. In six of the questions, the mailed respondents had a numerically significant (five
point or larger) performance rate. Finally, there was one specific trend: for all performance-related
questions for the air section of the survey, the mailed surveys indicated a better performance.
Based on these results, we can conclude that in general the mailed survey responses do not
provide a biased estimate of performance towards key environmental regulations, but there may be some
concerns for the air-related questions. One of the two cases where mail responders indicated significantly
better performance was an air-related question. Combined with the fact that mail responders indicated
better performance for all air-related questions, this may indicate some concern about the accuracy of the
mail responders for these questions.
There is one caveat to this analysis, however. The results are all based on small numbers of
respondents for each question. Specifically, the on-site questions had an average of 22 respondents with a
range of 6 to 38 respondents and the mailed survey had an average of 24 respondents with a range of 7 to
41 respondents.47 If we had more respondents, we may have found more significant differences.
Nevertheless, we expect that our results are representative enough to conclude that, except for the air
questions, the mail responders accurately depicted their performance.
47 These numbers differ from the total response for the surveys and from question to question, since not all questions
applied to all respondents.
-------
Page 84
Section VI: OECA's Experience with the Dillman Method
Table 6-11. Compliance Comparison for Metal Finishing Sector
Compliance Category
Wastewater Questions
Have Chrome Permits and Chrome
Treatment
Have Cyanide Permits and Cyanide
Treatment
Have Metal-Bearing Permits and Metal-
Bearing Treatment
Conduct Monitoring as Required by Permit
Conduct Reporting as Required by Permit
Conduct Sampling as Required by Permit
Hazardous Waste Questions
Comply with Time Limited Storage
Requirements
Comply with Labeling Requirements
Conduct Storage Area Inspections
Conduct Inspections with the Required
Frequency
Have Contingency Plans
Air Questions
Have Emission Controls (Relating to
Chromium) on Every Chromium Tank
Have one of the Recommended Control
Technologies
Both Use and Monitor the Control
Technology (Both are Required)
Have a Solvent Cleaning Machine with 5%
or Greater by Weight of a Regulated
Solvent and Know that the Solvent
Cleaning Standards Apply
Mailed
Percentage
On-Site
Observation
Percentage
Better
Performance
Statistically
Significance
Difference
87.0%
(n=23)
93.3%
(n=15)
93.5%
(n=31)
92.7%
(n=41)
87.8%
(n=41)
85.4%
(n=41)
70.0%
(n = 20)
90.0%
(n=10)
100.0%
(11=34)
84.2%
(n=38)
89.5%
(n=38)
89.5%
(n=38)
Mailed
Mailed
On-Site
Mailed
On-Site
On-Site
Yes
No
Yes
No
No
No
100.0%
(11=22)
77.3%
(n=22)
100.0%
(11=22)
86.4%
(n=22)
86.4%
(n=22)
95.5%
(n = 22)
100.0%
(11 = 22)
100.0%
(11 = 22)
95.5%
(n = 22)
86.4%
(n = 22)
Mailed
On-Site
Same
On-Site
Same
No
Yes
No
No
No
93.3%
(n=15)
100.0%
(11=15)
86.7%
(n=15)
71.4%
(n=7)
84.6%
(n=13)
92.3%
(n=13)
53.8%
(n=13)
66.7%
(n=6)
Mailed
Mailed
Mailed
Mailed
No
No
Yes
No
-------
Section VI: OECA's Experience with the Dillman Method
Page 85
Table 6-12. Summary of Compliance Question Results for Metal Finishing -
Comparison of Mailed Surveys and On-Site Observation Compliance Rates
Category
Questions where mailed survey participants had a higher
performance than on-site observations
Questions where mailed survey participants had a statistically
significantly higher performance than on-site observations
Questions where mailed survey participants had a higher
performance, five percentage points or more, than on-site
observations
Number
(Total Compared = 15)
8
2
6
Percentage of
Compliance
Questions
53%
13%
40%
Marinas
Tables 6-13 to 6-16 compare the compliance performance between the marinas that participated
in the two surveys. Table 6-17 summarizes the numbers and percentages of questions where mailed
respondents had (a) higher reported compliance, (b) statistically significant higher reported compliance,
and (c) a compliance rate five percentage points higher. As with the metal finishing analysis above, the
purpose is to compare the mailed survey compliance rates to the on-site observations compliance rates
under the assumption that the on-site observations would generate an accurate compliance rate. For each
compliance-related question, we perform a statistical test to determine if there was a significant difference
between the two samples.
In total, we compared 38 questions and found that in 29 cases (76 percent of the questions) the
mailed responders indicated higher compliance. For 23 of those questions, the mailed responders
indicated a statistically significant higher compliance rate and in 22 cases the mailed responders had
compliance rates five percentage points or higher. Among the nine cases where the on-site observations
resulted in higher compliance, only four were significant. Furthermore, for all of the hazardous waste
questions (10 questions total; see Table 6-13) and all of the hazardous materials questions (3 questions
total; see Table 6-15) the mailed responders indicated significantly higher compliance.
This analysis shows that the mailed responders indicated significantly higher compliance than the
on-site observations found. Thus, compliance-related questions from the mailed survey may not
accurately reflect compliance. As noted above in comparing the characteristics of the samples, however,
it appears that the two samples may be different, which may result in some discrepancies between the
-------
Page 86
Section VI: OECA's Experience with the Dillman Method
two modes. Nevertheless, the widespread differences between reported compliance by the mailed
responders and observed compliance during the site visits should call into question the validity of
obtaining reliable compliance information for this sector through a mailed survey.
Table 6-13. Hazardous Waste Compliance for Marinas
Performance Characteristic
Manifests for three years
Employee training in waste handling
Employee training in emergency procedures
Quantities calculated to determine size
Storage in labeled containers
Storage in dedicated area
Stored indoors or covered
Storage area with impervious floor
Storage area spill containment
Shipped with properly licensed transporter
Mailed
Percentage
80.5%
(n=87)
97.0%
(n = 100)
91.7%
(n = 96)
39.1%
(n=87)
96.8%
(n=93)
95.7%
(n=93)
95.7%
(n=93)
78.9%
(n = 90)
70.4%
(n=81)
94.1%
(n=85)
On-Site Visit
Percentage
78.2%
(n=55)
81.3%
(n = 64)
81.3%
(n = 64)
15.9%
(n = 63)
58.3%
(n = 60)
82.0%
(n = 61)
73.8%
(n = 61)
67.2%
(n = 61)
45.9%
(n = 61)
84.0%
(n=50)
Better
Performance
Mailed
Mailed
Mailed
Mailed
Mailed
Mailed
Mailed
Mailed
Mailed
Mailed
Statistically
Significant
Difference
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Table 6-14. Oil and Fuel Compliance for Marinas
-------
Section VI: OECA's Experience with the Dillman Method
Page 87
Performance Characteristic
All oil/fuel tanks less than 660 gallons
Above ground capacity less than 1,320 gallons
Below ground capacity less than 42,000 gallons
SPCC on site
SPCC signed by PE
SPCC posted in plain view at storage locations
Have above ground storage secondary
containment
Have above ground storage leak detection
Have below ground storage secondary
containment
Have below ground storage leak detection
Spill prevention procedures for receiving oil
from supplier
Spill prevention procedures for transferring
within facility
Spill prevention procedures for waste oil
disposal
Spill prevention procedures for fuel dispensing
Spill containment equipment
Mailed
Percentage
71.8%
(n= 103)
76.6%
(n=94)
96.6%
(n=89)
57.1%
(n=77)
27.0%
(n=63)
31.4%
(n=70)
77.6%
(n=58)
43.6%
(n=55)
82.9%
(n=35)
89.7%
(n=39)
77.0%
(n=61)
70.2%
(n=47)
82.9%
(n=82)
89.4%
(n = 66)
92.5%
(n=67)
On-Site Visit
Percentage
71.7%
(n=53)
80.8%
(n=52)
100.0%
(n=52)
5.9%
(n=51)
4.0%
(n=50)
2.0%
(n=51)
71.4%
(n = 21)
47.6%
(n = 21)
95.0%
(n = 40)
95.0%
(n = 40)
76.4%
(n=55)
47.4%
(n=57)
67.9%
(n=56)
96.1%
(n=51)
84.9%
(n=53)
Better
Performance
Mailed
On-Site
On-Site
Mailed
Mailed
Mailed
Mailed
On-Site
On-Site
On-Site
Mailed
Mailed
Mailed
On-Site
Mailed
Statistically
Significant
Difference
No
No
Yes
Yes
Yes
Yes
No
No
Yes
No
No
Yes
Yes
Yes
Yes
Table 6-15. Hazardous Materials Compliance for Marinas
-------
Page 88
Section VI: OECA's Experience with the Dillman Method
Performance Characteristic
On-site storage calculation
MSDSonfile
MSDS used in training
Mailed
Percentage
62.9%
(n=62)
90.7%
(n=97)
89.0%
(n=91)
On-Site Visit
Percentage
23.4%
(n = 64)
64.1%
(n = 64)
52.9%
(n = 68)
Better
Performance
Mailed
Mailed
Mailed
Statistically
Significant
Difference
Yes
Yes
Yes
Table 6-16. Storm Water Compliance for Marinas
Performance Characteristic
Required to have permit
Has required NPDES Permit
Ablative paints: prevent material release to
water
Ablative paints: prevent material release to land
Blasting, paint preparation and painting: prevent
release to water
Blasting, paint preparation and painting: prevent
release to land
Blasting, paint preparation and painting: prevent
release to protect employees
Engine fluids release contained
Fuels, solvents, and paints stored away from
drains
Fuels, solvents, and paints plainly labeled
Mailed
Percentage
27.8%
(n=72)
100.0%
(n=19)
53.0%
(n=83)
46.4%
(n=84)
89.0%
(n=82)
82.9%
(n=82)
100.0%
(n=79)
100.0%
(n = 100)
98.1%
(n = 104)
98.0%
(n=98)
On-Site Visit
Percentage
75.7%
(n = 70)
2.4%
(n = 41)
60.7%
(n = 61)
57.4%
(n = 61)
84.3%
(n=51)
84.3%
(n=51)
90.6%
(n=53)
96.6%
(n=58)
95.1%
(n = 61)
91.8%
(n = 61)
Better
Performance
NA[a]
Mailed
On-Site
On-Site
Mailed
On-Site
Mailed
Mailed
Mailed
Mailed
Statistically
Significant
Difference
Yes
Yes
No
Yes
No
No
Yes
Yes
No
Yes
[a] This is not purely a compliance question, but reflects the percentage that require a NPDES permit. Thus,
"better performance" does not apply in this situation. A significantly smaller percentage of mailed survey
respondents indicated that they did not require a NPDES permit compared to the site visit facilities. Thus, this is a
concern since the site visits are being used to validate the mailed survey responses.
Table 6-17. Summary of Compliance Question Results for Marinas -
-------
Section VI: OECA's Experience with the Dillman Method
Page 89
Comparison of Mailed Surveys and On-Site Observation Compliance Rates
Category
Questions where mailed survey participants had a higher
compliance rate
Questions where mailed survey participants had a
significantly higher compliance rate
Questions where mailed survey participants had a higher
compliance rate of five percentage points or more
Number
(Total = 38)
29
23
22
Percentage of
Compliance
Questions
76.3%
60.5%
57.9%
Summary
The two surveys (metal finishing and marinas) had different goals and therefore yielded different
results in regards to the reliability of mailed survey data. The metal finishing sector surveys asked
questions on awareness and general trends in environmental performance; for this sector the mailed
responses are not significantly different than the on-site observations. The marinas surveys, however,
asked specific behavioral and compliance questions; the two survey modes for this sector yielded very
different results.
The two surveys also asked for environmental performance data in different ways. The metal
finishing survey asked for information that EPA then used to evaluate environmental performance (e.g.,
"How long is the hazardous waste maintained on site?") The marina survey asked facilities to respond
more directly as to if they were in compliance (e.g., "Are all hazardous wastes shipped with a properly
licensed transporter?") Because the questions on the marina survey required recipients to directly self-
report compliance status, it is likely that the mailed marina survey would yield different results than the
mailed metal finishing survey (i.e., marinas would not want to make themselves "look bad").
-------
Page 90 Section VI: OECA's Experience with the Dillman Method
E. Lessons Learned
How to Improve Response Rates
Clearly define your target population.
It is extremely important to clearly define your target population at the beginning of the project.
Are you looking only for sites performing certain operations or processes? Are you looking only for a
specific media of release? For example, metal finishers are typically characterized as performing plating
and chemical conversion coating operations. However, this study also looked at facilities generating
RCRA wastes and air emissions from other operations. Therefore, facilities that didn't perform typical
metal finishing operations (e.g., painting facilities conducting surface preparation operations such as
solvent degreasing) may not have considered the survey as applicable to them.
Keep your target population and mailing list data sources as local as possible.
The geographic span of your target population can impact the source of mailing data that you use.
If you are targeting only a small subset of facilities, such as New England Marinas, then data can be
obtained at a local level. However, if you are sending out a national, or multi-regional survey, you are less
likely to have the resources to contact each locality for information. In these cases, a national database is
generally required, which likely will not provide as much detail as can be obtained from local data
sources. For example, for the metal finishing study EPA considered focusing on two cities, and using data
from publicly owned treatment works (POTWs) and wastewater permit databases to identify survey
recipients. In these cases, EPA would have been fairly certain that these facilities were in scope, since the
POTWs and permit databases have data on the types of operations performed. However, by focusing on
EPA Regions 1 and 5, the use of POTWs became unmanageable, so EPA used a national database (the
Harris Directory) that included very general information on products and SIC code, but not specific
information on the types of process performed. This likely increased the nonresponse and out-of-scope
rates.
-------
Section VI: OECA's Experience with the Dillman Method Page 91
For voluntary compliance assistance programs, evaluate if the industry is receptive to EPA.
As discussed at the beginning of the section, EPA conducted the surveys for sectors in two stages
of compliance assistance activity: 1) a sector for which EPA is beginning a compliance assistance effort
(marinas); and 2) a sector for which EPA has conducted several compliance assistance activities (metal
finishing). For both sectors, EPA and the regions coordinated with industry groups during project
development. For the marina sector, EPA Region 1 was in the initial stages of a compliance assistance
program, and had conducted outreach to the marinas. There were no other ongoing regulatory activities,
and the marinas were receptive to the compliance assistance programs. For the metal finishers, however,
EPA was in the midst of a contentious regulatory development program. Therefore, while EPA/OC
worked with the trade associations to foster cooperation, some metal finishers may not have participated
because of previous negative experiences with EPA.
Work with the trade associations and state compliance assistance personnel.
For both sectors, EPA worked closely with the trade associations and the state regulatory
personnel in the study regions to both develop the survey instruments and foster cooperation with the
industry. We believe that this communication helped improve the response rates in both sectors, and
helped overcome some of the hurdles discussed above with the metal finishing sector.
Make sure the timing is right.
Because of an upcoming EPA Region 1 compliance assistance workshop, one portion of the
marina survey mailout had to occur between the Christmas and New Year holidays when many
businesses shut down. Very few surveys were returned as a result of this initial mailing. The entire metal
finishing site visit effort was scheduled near the end of the calendar year when many businesses are at
their busiest. Several facilities told the contractor that they were too busy for a site visit, but would be
available at the start of the new year. These factors may have reduced the response rate. Timing your
efforts to avoid major holidays or the height of a business season can help to increase the response rate of
your project.
-------
Page 92 Section VI: OECA's Experience with the Dillman Method
How to Choose the Right Data Gathering Technique
What type of data do you need?
When choosing the data gathering technique, give careful thought to the type of data you wish to
gather. For the metal finishing survey, EPA focused on awareness of environmental requirements and
general environmental performance, but did not ask specific compliance information. As a result, the
mailed survey results closely matched the on-site observations. However, for the marina survey, EPA
asked more specific behavioral and compliance questions, and the results from the mailed survey did not
match the results from the on-site observations. Consequently, questions pertaining to general facility
information, regulatory awareness, general trends in environmental performance or customer satisfaction
can reliably be obtained from a mailed survey. Whereas, if you are interested in detailed compliance or
behavioral information, on-site observations are the more appropriate data gathering technique. Other data
gathering techniques are also available, such as telephone or e-mail surveys, which are discussed in
Section III.
Site visits provide a great deal of additional anecdotal information that can have enormous value
to those running the programs. Visited metal finishing facilities were very willing to discuss their
thoughts on current compliance assistance activities. Through one-on-one conversations, they often
became comfortable enough to openly discuss their compliance assistance preferences, make suggestions,
and even ask questions regarding areas of confusion. Also, as discussed previously, by visiting the
facility, the site-visit personnel were able to clarify areas of the survey instrument where the facility
personnel may have been confused. Very few returned mailed surveys provided any information that was
not specifically requested. When choosing the type of data gathering technique to use, you will need to
decide if anecdotal information will be helpful to your effort.
Finally, during site visits, site-visit personnel were able to provide compliance assistance tools to
the facility personnel. Many sites were very appreciative of the additional guidance and tools provided by
the site-visit personnel. Site visits also provide the opportunity to establish a working relationship with
the site. Heavily regulated industries, such as the metal finishing industry, may have a biased perception
of voluntarily working with EPA. A personal, non-enforcement based visit can help to reshape those ideas
and foster future cooperation.
-------
Section VI: OECA's Experience with the Dillman Method Page 93
What are your resources?
As shown in the results section above, the resources required vary greatly by the type of data that
you need. Mailed surveys can be used when collecting general facility information, regulatory awareness,
general trends in environmental performance or customer satisfaction type information. Site visits, while
more costly, can provide detailed compliance information, additional anecdotal information, and can help
to improve industry relations. Resources can dictate both the data gathering technique and the scope of
the project. If resources are limited but detailed compliance information is required, a site visit program o
f reduced scope (i.e., non-statistical) may be the best solution.
-------
Page 94 Section VI: OECA's Experience with the Dillman Method
Figure 6-1. Metal Finishing Performance Evaluation Survey
Section 1: Wastewater Treatment
1. Does your site generate chromium-bearing wastewater?
DYes DNo
2. Does your site generate cyanide-bearing wastewater?
DYes DNo
3. Does your site generate metal-bearing wastewater?
DYes DNo
4. Which types of wastewater treatment are used on site (check all that apply):
None D
Equalization/Neutralization D
Chromium Reduction D
Cyanide Destruction D
Oil Skimming/Chemical Emulsion Breaking D
Chemical Precipitation and Sedimentation D
Complex/Chelated Metals Treatment D
Sand/Multimedia Filtration D
Membrane Filtration (Microfiltration, Ultrafiltration) D
Ion Exchange D
Sludge Dewatering D
Other:
5. Do you have a wastewater discharge permit?
DYes DNo
6. Which activities listed below do you perform for your wastewater discharge?
Monitoring (e.g., flow, pH, ORP) D
Sampling of wastewater for specific pollutant parameters D
Reporting D
7. Have you received any compliance assistance to reduce the amount of wastewater your facility produces or
to reduce the amount of metals in your wastewater?
DYes DNo
8. If yes, from whom? And was it helpful?
9. Are there any wastewater regulations that you feel you would like additional assistance with? If so, what
are they?
10. What sources do you use for information on wastewater environmental regulations and pollution
prevention? (Check all that apply)
Internet D Trade/professional organizations D
Industry/professional journals D Environmental consultants D
Colleagues/supervisor D Government environmental agencies D
-------
Section VI: OECA's Experience with the Dillman Method Page 95
Section 2: Hazardous Waste Handling
11. Does your facility generate hazardous waste (e.g. F006 wastewater treatment sludge)?
DYes DNo
If answer to Question 11 is no, skip to Question 26.
12. Is hazardous waste generated in excess of 100 kg/month?
DYes DNo
13. How long is the hazardous waste maintained on-site?
days
14. Are you aware of the hazardous waste labeling requirements for drums?
DYes DNo
15. Do they apply to your drums?
DYes DNo
16. If not, why not?
17. Are you aware of the hazardous waste manifest records requirements?
DYes DNo
18. If yes, do these hazardous waste manifest records apply to your facility?
DYes DNo
19. Do you perform inspections of your hazardous waste storage area?
DYes DNo
20. If yes, how often are inspections performed:
21. Do you have a hazardous waste contingency plan?
DYes DNo
22. Have you received any compliance assistance to help you with the identification and/or handling of
hazardous waste treatment sludge?
DYes DNo
23. If yes, from whom? And was it helpful?
24. Are there any hazardous waste handling regulations that you feel you would like additional assistance with?
If so, what are they?
25. What sources do you use for hazardous waste information on environmental regulations and pollution
prevention? (Check all that apply)
Internet D Trade/professional organizations D
Industry/professional journals D Environmental consultants D
Colleagues/supervisor D Government environmental agencies D
-------
Page 96 Section VI: OECA's Experience with the Dillman Method
Section 3: Air Emissions
26. Does your site perform chromium electroplating or chromic acid anodizing operations?
DYes DNo
If answer to Question 26 is 'NO', skip to Question 34.
27. How many chromium electroplating or chromic acid anodizing tanks do you operate (excluding rinses)?
tanks
28. Are you aware of emissions control requirements for your tanks?
DYes DNo
29. If yes, how many of your tanks use some type of emission control?
tanks
30. Do you use and monitor any of the following technologies on your chromium electroplating or chromic acid
anodizing solutions (check all that apply):
Use Monitor
Composite mesh-pad system D D
Fiber-bed mist eliminator D D
Packed-bed scrubber D D
Chemical fume suppressant in plating baths D D
Other:
31. Do you have a copy of the EPA Guidebook on How to Comply with the Chromium Electroplating and
Anodizing NESHAP?
DYes DNo
32. Have you ever received any assistance to help you comply with EPA's Chromium MACT standards?
DYes DNo
33. If yes, from whom? And was it helpful?
34. Do you own or operate a solvent cleaning machine? DYes DNo
If answer to Question 34 is 'NO', skip to Question 44.
35. Does the solvent cleaning machine use a solvent containing 5 % or more by weight of any one or
combination of the following halogenated solvents (check all solvents that are used):
Methylene Chloride D 1,1,1-trichloroethane D
Perchloroethylene D Carbon tetrachloride D
Trichloroethylene D Chloroform D
36. Which of the following solvent cleaning machines are used?
Batch Cold Machine D In-line Cold Machine D
Batch Vapor Machine D In-line Vapor Machine D
37. Are you aware of the EPA emissions standards for solvent cleaning machines?
DYes DNo
-------
Section VI: OECA's Experience with the Dillman Method Page 97
38. If yes, do these standards apply to your machines?
DYes DNo (Why not? )
39. If these standards apply, what method does your site use to comply with emission standards for solvent
cleaning machines?
Equipment compliance option D
Overall emission limit D
Not certain D
40. Do you have a copy of the EPA Guidance Document for Halogenated Solvent Cleaner NESHAP?
DYes DNo
41. Have you used the EPA's Halogenated Solvent Cleaner Rule Assistance Website?
DYes DNo
42. Have you ever received any assistance to help you comply with EPA's Solvent MACT standards?
DYes DNo
43. If yes, from whom? And was it helpful?
44. Are there any air regulations that you feel you would like additional assistance with? If so, what are they?
45. What sources do you use for information on air environmental regulations and pollution prevention?
(Check all that apply)
Internet D Trade/professional organizations D
Industry/professional journals D Environmental consultants D
Colleagues/supervisor D Government environmental agencies D
Section 4: Toxics Release Inventory (TRI) Reporting
46. Are you aware of the EPA Form R (TRI) for Toxic Chemical Release Reporting?
DYes DNo
47. Is your facility exempt from TRI reporting?
DYes DNo If yes, why?
48. Have you ever received any assistance to help you complete your TRI reporting?
DYes DNo
49. If yes, from whom? And was it helpful?
50. Are there any TRI reporting requirements that you feel you would like additional assistance with? If so,
what are they?
51. What sources do you use for information on TRI Reporting? (Check all that apply)
Internet D Trade/professional organizations D
Industry/professional journals D Environmental consultants D
Colleagues/supervisor D Government environmental agencies D
-------
Page 98 Section VI: OECA's Experience with the Dillman Method
Section 5: General
52. Do you have internet access?
DYes DNo
53. Have you used the National Metal Finishing Resource Center internet site (nmfrc.org)?
DYes DNo
If answer to Question 53 is no, skip to question 57.
54. If yes, how do you rate the usefulness of the assistance provided?
DVeryGood DGood DFair DPoor
55. What type of information have you obtained from the NMFRC (check all that apply)?
Compliance Information D
Environmental Regulations D
Process information D
56. What types of actions have you taken as a result of information you obtained on NMFRC (check all that
apply)?
Changed process D Modified reporting procedures D
Implemented pollution prevention practices D None D
Implemented pollution control D Other:
Obtained permit D
57. Would you find the following types of compliance assistance helpful?
Yes No
Internet based training/guidance D D
Workshops/Training D D
CD ROM D D
On-Site Technical Assistance Visits D D
Inspections D D
Mailed Fact Sheets/Guidance D D
58. Are you aware of the National Metal Finishing Stategic Goals Program?
DYes DNo
59. What is the number of full-time equivalent employees working at your site?
-------
Section VI: OECA's Experience with the Dillman Method Page 99
Figure 6-2. MARINA CHECKLIST
US Environmental Protection Agency, New England
For All Questions, Please Use:
Y:Yes N: No DK: Don't Know NA: Not Applicable
Facility Type
1. a. How many employees work at the facility during peak boating recreational boating season?
Full-time Part-time
b. How many boats (capacity) are moored at the facility? Docked? Stored on land?
2. a. Are maintenance or repair operations performed at the facility?_
b. If yes, are these primarily customer do-it-yourself activities?
Hazardous Waste
3. a. Are manifests documenting hazardous waste shipments kept, going back at least 3 years?.
b. Do employees receive training in: Proper handling of wastes? Emergency procedures?
c. Are quantities of hazardous waste generated by the marina calculated each month, to determine what size
generator the marina is?
d. Are all hazardous wastes stored: In labeled containers? In a dedicated storage area?
Indoors or covered? In an area with an impervious floor?
With storage area spill containment?
e. Are all hazardous wastes shipped with a properly licensed transporter?
Oil and Fuel
4. a. Is oil (including motor fuel) stored above ground in any single tank with over 660 gallons capacity?
Above ground in total aggregate capacity of over 1320 gallons?
Below ground in total aggregate capacity of 42000 gallons?
b. Is a Spill Prevention, Control and Countermeasure plan (SPCC):
On site? Signed by a registered professional engineer?
Posted in plain view at oil storage locations?
5. a. Does above ground oil storage (including piping system) have: Secondary containment?
Leak detection?
b. Does below ground oil storage (including piping system) have: Secondary containment?
Leak detection?
c. Are spill prevention procedures in place for:
Receiving oil from a supplier? Transferring oil within the facility?
Waste oil disposal?
6. a. Does the facility have spill prevention procedures in place for fuel dispensing?
-?
Overfill alarm? Automatic shutoff? Fuel collars to capture splash/drips?_
Employee monitoring of fueling? Other?
b. Is equipment available and procedures in place to contain a spill at the dispenser location?
c. How would you evaluate the effectiveness of the methods you are using to prevent releases?
Excellent Good Fair Poor
-------
Page 100 Section VI: OECA's Experience with the Dillman Method
Hazardous Materials
7. Has the amount of each hazardous material stored on-site been calculated (including motor fuel in above-ground
systems of greater than 10,000 pounds capacity) to determine if reporting to the Local Emergency Planning
Committee is necessary?
8. Are Material Safety Data Sheets (MSDS) for all hazardous chemicals kept on file?
Used for training all employees handling hazardous chemicals?
Storm Water
9. a. Is a National Pollutant Discharge Elimination System (NPDES) Storm Water Permit required of this
facility?
b. Does the marina have a NPDES Storm Water Permit?
10. a. When pressure washing boats coated with ablative paints, are any efforts undertaken to prevent removed
material from releasing to water? from contaminating land?
b. Are blasting, other paint preparation and painting activities contained or controlled to prevent abrasives,
paint chips, and overspray from being released to the water? to land? to protect employees?
c. Are all engine fluids promptly transferred from parts, drip pans, used filters and other containers to closed
receptacles for disposal or recycling?
d. Are fuels, solvents and paints stored in a protected, secure location, away from drains?
Plainly labeled?
11. Has the marina made structural changes to minimize surface water runoff? Berming?
Vegetation? Riprap? Drains? Placement of filters in drains?
Other?
Other
12. Has the facility switched to alternative materials or products to reduce toxicity or other hazards to Health,
safety or the environment? Safer paint stripping? Safer painting?
Safer MSD odor chemicals? Dust collection? Phosphate free cleaners? Biodegradable
cleaners? Safer antifreeze? Other?
13. Does the facility have a sewage pumpout system? Onshore Boat How many gallons
(approx) do you pump out per week? Do you use Clean Vessels Act funds?
-------
APPENDICES
-------
APPENDIX A
-------
Appendix A Page A-l
PERFORMANCE PROFILE FOR
EPA's ENFORCEMENT AND COMPLIANCE ASSURANCE PROGRAM
Impact on Environmental or Human Health Problems
Measured through annual evaluation studies of selected EPA objectives.
Effects on Behavior of Regulated Populations
Levels of Compliance in Regulated Populations
Set 1. Rates of noncompliance for ~
a) fully-inspected populations
b) self-reported compliance information
c) populations targeted for special initiatives
d) priority industry sectors
Environmental or Human Health Improvements by Regulated Entities
Set 2. Improvements resulting from EPA enforcement action
Set 3. Improvements resulting from compliance assistance tools and initiatives
Set 4. Improvements resulting from integrated initiatives
Set 5. Self-policing efforts by using compliance incentive policies
Responses of Significant Violators
Set 6. Average number of days for significant violators to return to compliance or enter
enforceable plans or agreements
Set 7. Percentage of significant violators with new or recurrent significant violations within two
years of receiving previous enforcement action
Enforcement and Compliance Assurance Activities
Monitoring Compliance
Set 8. Number of inspections, record reviews, responses to citizen complaints, and
investigations conducted
Enforcing the Law
Set 9. Number of notices issued, civil and criminal actions initiated and concluded, and self-
policing settlements concluded
Providing Assistance and Information
Set 10. Facilities/entities reached through ~
a) compliance assistance tools and initiatives
b) distribution of compliance information
Building Capacity
Set 11. Capacity building efforts provided to state, local, or tribal programs
-------
APPENDIX B
-------
Appendix B Page B-l
OFFICE OF COMPLIANCE
GUIDANCE
ON THE NEED FOR
INFORMATION COLLECTION REQUESTS
(ICRS)
FOR THE EVALUATION OF
COMPLIANCE ASSISTANCE ACTIVITIES
SEPTEMBER 1997
Produced by the Office of Compliance Regional Compliance Assistance Work Group
Lynn Vendinello, Work Group Chair
Updated 2007
Hans Scheifele
-------
Page B-2 Appendix B
Federal agencies are generally required, by the Paperwork Reduction Act (PRA), to receive Office of
Management and Budget approval prior to collecting substantially similar information from ten or more
non-Federal respondents. A "collection of information" means the obtaining or soliciting of information
by an agency by means of identical questions, or identical reporting or record keeping requirements,
whether such collection of information is mandatory, voluntary, or required to obtain a benefit. This
includes any requirement or requests to obtain, maintain, retain, report or publicly disclose information. 5
CFR § 1320.3(c)
There are exceptions to this rule and depending on your particular situation, your compliance
assistance evaluation task may or may not fall within an exception. This guidance will help determine
whether or not an Information Collection Request (ICR) is necessary for your task. You may also contact
David Coursen in the Cross-Cutting Issue Division of the Office of General Counsel at 202-564-0781 to
assist you with individual questions.
What is the Paperwork Reduction Act?
The PRA is a law (PL 104-13) originally enacted by Congress in 1980, reauthorized in 1986 and
revised in 1995, that essentially attempts to minimize the Federal paperwork burden on the public.
Section 3501 of the law clearly states the eleven purposes of this Act.
"§3501. Purpose
The purposes of this chapter are to-
1. Minimize the Federal paperwork burden for individuals, small businesses, State and local
governments, and other persons resulting from the collection of information by or for the Federal
Government;
2. Ensure the greatest possible public benefit from and maximize the utility of information created,
maintained, used, shared and disseminated by or for the Federal Government;
3. Coordinate, integrate, and to the extent practicable and appropriate, make uniform Federal
information resources management policies and practices as a means to improve the productivity,
efficiency and effectiveness of Government programs, including the reduction of information
collection burdens on the public and the improvement of service delivery to the public;
4. Improve the quality and use of Federal information to strengthen decision making, accountability, and
openness in Government and society;
5. Minimize the cost to the Federal Government of the creation, collection, maintenance, use,
dissemination, and disposition of information.
6. Strengthen the partnership between the Federal Government and State, local, and tribal governments
by minimizing the burden and maximizing the utility of information created, collected, maintained,
used, disseminated, and retained by of for the Federal Government;
7. Provide for the dissemination of public information on a timely basis on equitable terms, and in a
manner that promotes the utility of the information to the public and makes effective use of
information technology;
8. Ensure that the creation, collection, maintenance, use, dissemination, and disposition of information
by or for the Federal government is consistent with applicable laws, including laws relating to--(a)
privacy and confidentiality,
-------
Appendix B Page B-3
9. Ensure the integrity, quality and utility of the Federal statistical system;
10. Ensure that information technology is acquired, used, and managed to improve performance of
agency missions, including the reduction of information collection burden on the public; and
11. Improve the responsibility and accountability of OMB and all other Federal agencies to Congress and
to the public for implementing the information collection review process, information resources
management, and related policies and guidelines established under this chapter."
What is an ICR?
An Information Collection Request (ICR) is a document submitted by federal agencies to OMB in
order to obtain approval of an information collection and/or a reporting and record keeping requirement
that falls under the purview of the PRA. The ICR must receive OMB approval prior to the initiation of the
information collection.
The term "collection of information" according to the Paperwork Reduction Act of 1995 (PL 104-
12(8.244)) means: "(A) the obtaining, causing to be obtained, soliciting, or requiring the disclosure to
third parties or the public, of facts or opinions by or for an agency, regardless of form or format, calling
for either-
"(i) answers to identical questions posed to, or identical reporting or record keeping requirements
imposed on, ten or more persons, other than agencies, instrumentalities, or employees of the United
States; or
"(ii) answers to questions posed to agencies, instrumentalities, or employees of the United States
which are to be used for general statistical purposes."
For guidance on how to complete an ICR, you can visit the "EPA's Information Collection Request
(ICR) Homepage" on the EPA Intranet site at or contact the current
OECA desk officer in the Office of Environmental Information at 202 566-1457.
When is an ICR Needed?
An ICR is generally required for any activity involving the collection of identical information from
ten (10) or more non Federal respondents in any twelve month period. ICRs may be approved for up to a
three-year period and can be extended through subsequent approval requests each fiscal year. An
approved ICR is required as long as the activity continues.
Examples of information collection activities that commonly require an ICR:
• Information requirements in a rule (e.g. reporting, record keeping, waiver provisions).
• Other information collection activities (e.g. studies, surveys, application forms, audits,
standardized data collection activities).
Certain activities are not subject to the Act. For example:
• An ICR is not required when the information is collected during the conduct of a criminal or civil
enforcement action.
-------
Page B-4 Appendix B
• An ICR is not needed when the collection falls under one of the categories of items that OMB has
concluded do not generally meet the definition of "information" contained in 5 CFR §1320.3(h).
Many of the compliance assistance activities that the Office of Compliance is currently undertaking as
well as those of the compliance assistance programs in the regions fall under one of the categories. To
assist in the determination of the need for an ICR, the following examples of scenarios that do and do not
require ICRs.
ICR APPLICABILITY SCENARIOS
Category A: Web -Sites:
Scenario One: I am establishing a web site for my regional compliance assistance program (or for a
compliance assistance center) and would like to establish a "comments" button or "feedback" feature.
Response: Generally, no ICR would be required for this activity. According to OMB, "an
undifferentiated 'suggestion box' format—such as one requesting 'ideas, comments, suggestions, or
anything else you would like to tell us,' or one asking, 'if you experience any technical problems with our
site, or have any suggestions for improving it, please let us know'-are not considered to be 'identical
questions'48. Such general solicitations of comments from the public do not require OMB clearance.
However, should the agency request specific information from web site users, OMB approval would be
required as explained in Scenario Two.
Scenario Two: I would like to put an on-line survey on my web site to determine what features of the
web site are most useful and to ask for suggestions for improving the web site.
Response: The fact that your survey is on-line does not affect the decision as to whether or not the survey
requires OMB clearance. What will affect whether or not the survey requires an ICR is the nature of your
questions. According to OMB guidance, if your questions are non-identical then you will not need OMB
clearance. Identical questions ask each respondent to supply the same level of information on the same
subject. For example, they often supply a specific set of answers for the user to select from. Non-identical
questions are non-specific and allow the responder to apply "facts or opinions" of their own choosing
without any direction from the government. In addition, if your survey is primarily for the purposes of
assessing customer satisfaction with your web site, you may want to consider using an existing ICR
Clearance for customer satisfaction. If your survey attempts to get at behavioral changes and/or
compliance improvements, then you may need a separate clearance again depending on the nature of your
questions. In general, if you feel that your questions are non-identical you may want to ensure a certain
degree of brevity with respect to your survey so that your survey does not appear to follow a plan of
inquiry.
Scenario Three: I would like to ask the users of my web site to identify themselves by name or by
category (e.g. auto service repair shop, car dealer, consultant).
Response: No ICR would be required for this activity. According to 5 CFR 1320.3 (h)(l), this category
of an inquiry is not deemed to constitute an information collection and therefore would not require
clearance. The Paperwork Reduction Act states that "Affidavits, oaths, affirmations, certifications,
receipts, changes of address, consents, or acknowledgment," do not constitute information. Merely,
asking users to identify themselves by name or category is a request for an "acknowledgment" not
generally subject to the PRA.
8"The Paperwork Reduction Act of 1995: Implementing Guidance", February 3, 1997, pg. 17
-------
Appendix B Page B-5
Category B: Workshops/Seminars/Training Sessions
Scenario Four: I am planning to hold a compliance assistance workshop for air permits. This workshop
is open to anyone who would like to attend (with limits on total numbers able to physically attend). After
the workshop is over, I would like to hand out a voluntary questionnaire that asks the attendees questions
such as: a) Has this workshop provided information that will help you improve your ability to comply
with environmental regulations?
Response: No ICR would be required for this activity. Exemption #8 of the Paperwork Reduction Act
that states that, "facts or opinions submitted in connection with public hearings or meetings"49 would
apply to this scenario. To provide for more certainty of this exemptions application, it would be best to
provide a Federal Register notice making it clear that the workshop is open to all interested members of
the public. A second-best option would be to adopt an open-door policy with respect to the workshop so
that no one would be excluded (except for obvious space limitations) from attending. In addition, you
could also conduct an on-the-spot evaluation of the workshop, since category #8 would most likely apply
to that activity, as well. You could also send a follow-up questionnaire within a short time period
following the seminar (e.g. one week).
Scenario Four A: My compliance assistance program has funding for four seminars this year. We would
like to determine the topics that would be of the greatest interest to our clients, so we would like to mail
out a voluntary questionnaire that lists potential seminar topics.
Response: An ICR would probably not be required for this activity. Category #8 of the PRA would
apply to this scenario as well. OMB guidance explains that, "included in this category are questions
which ask the proposed participants to identify themselves and the topic(s) about which they desire to
speak."50 Your request for topics to be discussed is similar to asking for a request to speak on a particular
topic. Further, the requested items are "in connection with" the public workshop and category #8 appears
to apply to such inquiries.
Scenario Four B: After the completion of a workshop, we would like to send a follow-up survey out
which asks questions about behavioral changes that resulted from attendance at the workshop.
Response: In this scenario you would probably need OMB clearance, especially if there was a significant
time delay before the survey was mailed out because the information collected would no longer pertain
directly to the public meeting that was held. If you were looking for behavioral changes that facilities
later adopted after the workshop had educated them about environmental requirements you will need an
ICR, for example, this information request is not directly related to evaluating the immediate impact of
the workshop (e.g., their satisfaction with the workshop and their improved awareness/understanding of
requirements).
Scenario Five: I will be holding a printing compliance training workshop that will be made generally
available to printers. I would like to administer a "test" before and after the training to determine if
understanding of environmental requirements changes as a result of the training.
Response: No ICR would be required for this activity. Category #7, "Examinations designed to test the
aptitude, abilities or knowledge of the persons tested and the collection of information for identification
495 CFR 1320.3 (h) (8)
50"The Paperwork Reduction Act of 1995: Implementing Guidance", February 3, 1997, pg 26
-------
Page B-6 Appendix B
or classification in connection with such examinations,"51 would apply to this scenario. The nature of your
test should be with respect to their knowledge of the subject matter at hand. If you wish to use the test to
collect socioeconomic information about the respondents, an ICR will probably be required.
Scenario Six: My office has given a grant to a state to develop a compliance guide that integrates federal
and state rules for metal finishers in my state. One of the criteria for awarding the grant was that the
grantee have a component for program evaluation. The grantee plans to include a comment card in the
compliance guide that would get mailed back to the state office.
Response: According to OMB guidance, "In general, collections of information conducted by recipients
of Federal grants do not require OMB approval. On the other hand, an agency is the sponsor of a
collection of information... if the grant recipient is: 1) collecting information at our specific request; and/or
2) the terms and conditions of the grant require that the we specifically approve the collection of
information or the collection procedures."52 One can ask for a program evaluation component of a grant
proposal and/or measures of success; however, we can not ask that a particular survey method be used
without getting an ICR approved.
If, however, the award is a cooperative agreement, then the agency is considered a sponsor of the
information and all of the PRA restrictions on information collection would apply.
Category C: Mailed or Phoned Surveys
Scenario Seven: An EPA employee or contractor would like to know how many states have a small
business policy and plans to call them to ask for a copy of their policy, if they have one.
Response: No ICR would be necessary to conduct this activity. Category #2 of the Paperwork Reduction
Act states that the request for "samples of products or any other physical objects"53 does not constitute a
collection of information. According to OMB, this category "includes requests for information that is
already available in a form suitable for distribution and is provided in that form to all requesters. (This
request is a collection of information if the information has to be compiled or if it is not provided to any
person who requests it)."54
Scenario Eight: An EPA employee or contractor would like to follow up with those states that have sent
us a copy of their small business policy to ask specific questions about their individual state policies.
Response: No ICR would be required for this activity. Since you will be asking each of the states
questions that pertain only to their specific policy and not identical questions of each state, Category #6 of
the PRA would apply. Category #6 of the PRA states that "a request for facts or opinions addressed to a
single person"55 does not constitute a request for information. However, if EPA asked the same questions
following a plan to more than ten states, the PRA would apply.
Scenario Nine: My EPA program would like to ask states to voluntarily answer a survey that asks them
to quantify the benefits of their compliance assistance program.
515 CFR 1320.3(h)(7)
52IBID. pg 14
535 CFR 1320.3(h)(2)
54"The Paperwork Reduction Act of 1995: Implementing Guidance," draft, February 3, 1995 pg. 24
555 CFR 1320.3(h)(6)
-------
Appendix B Page B-7
Response: It is important to understand that the PRA applies not only to industry and individuals but also
to requests for information from states and local governments. Further, the fact that the survey is
voluntary does not mean that the PRA does not apply. In this case OMB clearance would be required
because you are asking identical questions and are directing them to specific entities.
-------
APPENDIX C
-------
Appendix C Page C-l
How to Obtain Clearance
for
Regional Compliance Assistance Evaluation Surveys
Under the Generic ICR 1860.03
OMB Control # 2020.0015
Updated July 2007
U.S. Environmental Protection Agency
Office of Compliance
Office of Enforcement and Compliance Assurance
-------
Page C-2 Appendix C
Table of Contents
Steps to Developing the Clearance Package in a Glance 1
The Memos 2-3
Request for Approval of Information Collection Activity 4-5
The Survey 6-7
Frequently Asked Questions
Question 1: When do I need to use the generic compliance assistance ICR? 8
Question 2: Is an ICR required for all surveys? 8
Question 3: How do I receive approval for my survey, if it meets the conditions outlined above? . . 8
Question 4: How Long Will The Process Take? 8
Question 5: What are the new requirements in the latest ICR? 9
Question 6: Can I Send a List of Survey Questions That I Plan to Use over and over Again at
Workshops, Training Seminars, or On-site observations to OMB for Approval? 10
Question 7: What Do I Have to Do to My Survey to Show That it Has Been Approved by
OMB? 10
Question 8: Is There a Standard Cover Memo for the Clearance Package and, How Do I Calculate the
Necessary Burden Estimates? 10
-------
Appendix C Page C-l
Steps to Developing the Clearance Package in a Glance
• Write a brief memo from YOUR program office to the current, OECA Desk Officer
From: Your Name
Office Name
EXAMPLE:
To: Carolyn Scully, OECA Desk Officer
Collection Strategies Division
Office of Environmental Information
In the memo include:
Survey title
Originating office, point of contact with phone number
State whether your ICR is under the 10 day review (no behavior change questions) or the 30 day
review (survey has behavior change questions)
Number of expected survey respondents
Cost and Burden estimate
• Write a brief memo from the OECA Desk Officer in at EPA to the current OIRA Desk Officer
for OMB.
EXAMPLE:
From: Carolyn Scully, OECA Desk Officer
Collection Strategies Division
Office of Environmental Information
To: Rob Johansson, OIRA Desk Officer
Office of Management and Budget
In the memo include:
Survey title
Originating office, point of contact with phone number
State whether your ICR is under the 10 day review (no behavior change questions) or the 30 day
review (survey has behavior change questions)
Number of expected survey respondents
Cost and Burden estimate
• Develop the Request for Approval of Information Collection Activity
I. Background
II. Survey Purpose and Description
III. Survey Methodology and Use of Results
IV. Respondents Burden
V. Agency Burden
• Include a Copy of the Survey - remember to include the reporting burden statement at the end.
EXAMPLES AND INSTRUCTIONS
FOR ALL THE DOCUMENTS FOLLOW
-------
Page C-2 Appendix C
EXAMPLE Memo One
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
REGION I
1 CONGRESS STREET, SUITE 1100
BOSTON, MASSACHUSETTS 02114-2023
MEMORANDUM
DATE: July 17, 2000
SUBJ: Request for OMB Approval of a Survey for Viewers of the Video RCRA Compliance for
Metal Finishers.
FROM: Linda Darveau
New England Environmental Assistance Team
TO: Carolyn Scully, OECA Desk Officer
Collection Strategies Division
Office of Environmental Information
I am the EPA project officer on EPA Grant # X991962 to the Northeast Waste Management Official
Association, which is producing a 22-minute videotape for metal finishers entitled RCRA Compliance for
Metal Finishers. I would like to issue a mail-back survey to assess viewers' opinions of the video. Since
this survey contains behavioral change questions, it will necessitate the maximum 30-day OMB review
under the generic ICR.
I plan to offer the video free of charge by advertisement to approximately 1500 metal finishers and other
business and government contacts in New England. I estimate that we will distribute 500 videos. Each
video will be accompanied by a postage-paid return survey to assess how informative and useful the
viewers find the video. Based on a 20% response rate, I expect to receive 100 responses. This, in
combination with the brevity of the survey, means that the burden to both EPA and the public will be low.
If you have any questions or concerns, please call me at 617-xxx-xxxx.
Attachments (2)
CC: Hans Scheifele
-------
Appendix C Page C-3
EXAMPLE Memo Two
MEMORANDUM
SUBJECT: Review of Compliance Assistance Regional Program Evaluation Survey,
ICRNo. 1860.03 (OMB 2020-0015)
FROM:Carolyn Scully, OECA Desk Officer
Collection Strategies Division
Office of Environmental Information
TO: Rob Johansson, OIRA Desk Officer
Office of Management and Budget
As a condition of OMB approval for the generic Regional Compliance Assistance Program
Evaluations ICR, No. 1860.03, Expiration date March, 2008, OMB Control No. 2020-0015, EPA agreed
to submit each specific questionnaire covered by this clearance to OMB for review. Attached for your
review and approval is a Region 3 survey for evaluating the results of a mailing. The purpose of this
survey is to assess the effectiveness of the mailings in helping facilities to comply, informing them about
EPA's policy for waiving or reducing penalties for disclosed violations, and determining possible areas
for EPA improvement. Since this survey contains behavioral change questions, it will necessitate the
maximum 30-day OMB review under the generic ICR.
Note: EPA will not have access to names or addresses of facilities surveyed and will not use the
information for inspection or enforcement purposes. There will be no way to link participant responses to
specific facilities or individuals.
Your comments and suggestions would be much appreciated. For comments or questions on the
survey, please contact Janet Viniski at 215-xxx-xxxx.
If you have any questions about the ICR or this survey submission, please contact me at 202-566-
1457.
Thank you for your cooperation in this matter.
Attachments (1)
-------
Page C-4 Appendix C
EXAMPLE
Request for Approval of Information Collection Activity
I. Background:
Under EPA grant X991962 to NEWMOA, EPA Region I is producing a 22-minute videotape for metal
finishers entitled RCRA Compliance for Metal Finishers in the Northeast. The video is designed to
provide basic up-to-date information on RCRA Compliance to small facilities whose personnel have
difficulty sparing the time to attend workshops and read documents. The video includes information on
both regulatory requirements and pollution prevention. The video will convey this information in an
appealing, easy-to-digest manner by showing narrated views of real processes, focusing on the most
common violations found by both state and EPA inspectors. The video will be offered free of charge by
advertisement to 1300 metal finishers and other business and government contacts in New England. I
estimate that we will distribute 500 videos. Each video will be accompanied by a postage-paid return
survey to assess how informative and useful the viewers find the video.
II. Survey Purpose and Description:
I have drafted a postage-paid mail-back survey to accompany each video that is distributed. The purpose
of the survey is to assess how informative and useful the video is to members of the intended audience in
helping them understand and make plans to act upon pollution prevention opportunities. EPA will use this
information to improve future videos for this and other sectors and to choose appropriate outreach
methods (videos versus documents versus workshops, etc).
III. Survey Methodology and Use of Results
Each video requested and mailed out will be accompanied by the survey. Of the 1300 metal finishers and
other business and government contacts in New England to whom we will advertise the video free-of-
charge, I estimate that 500 will request a video. Based on a 20% response rate I expect to receive 100
responses. Each survey will be printed as a postage-paid self-mailer so that respondents do not have to
provide envelopes or postage. EPA Region I will be the mailing address to which the surveys are
returned, and Linda Darveau, the EPA contact for this project, will compile the results and provide them
to interested parties- EPA and state pollution prevention assistance programs, Metal Finishing trade
associations, and others.
IV. Respondents' Burden
Number of survey recipients: 500
Number of respondents: 100
Minutes per response: 10
Cost per hour: $82.74 ($39.40 + 110%)*
Total burden hours: 17 hours
Total burden dollar cost: $1,379.00
These rates are from the United States Department of Labor, Bureau of Labor Statistics, September 2004,
"Table 2: Civilian Workers, by occupational and industry group." These rates are from column 1 "Total
compensation ." The rates have been increased by 110% to account for the benefits packages available to
those employed by private industry.
-------
Appendix C Page C-5
V. Agency Burden
Number of survey recipients: 500
Number of respondents: 100
Minutes spent compiling paper responses, per response: 10
Burden hour subtotal for paper responses: 16
Additional hours to summarize and communicate all results: 15
Burden hour total: 32
Cost per hour: $49.29
(Based on a 2007 GS 13/01 salary of $37.91/hr; with 30% overhead the hourly rate is $49.29. This is
the estimate used in the EPA / OECA generic ICR.)
Total burden dollar cost: $1,577.28
-------
Page C-6 Appendix C
EXAMPLE Survey
OMB control number 2020-0015 Expires 3/31/08
1. About how often do you visit ?
- This is the first time
- a few times per year
- 1-2 times per month
- once a week or more
- whenever the need arises
2. helps me understand the environmental regulations that apply to .
- Strongly Agree
- Agree
- Neutral
- Disagree
- Strongly Disagree
- Have never used the for this purpose
3. What actions(s) have you taken to improve environmental practices, in whole or in part, due to
information you have found through the : (Select all that apply)
- contacted a vendor
- changed the handling of waste or emission
- performed a self audit
- contacted a regulatory agency
- changed a process or practice
- obtained a permit or certification
- took an action other than listed above to comply with a
- regulatory requirement
- Identified a pollution prevention opportunity
- other:
- no action was taken
4. Please identify whether you reduced, treated, or eliminated pollution as a result of use:
(select all that apply)
- reduced or eliminated waste (solid/hazardous)
- reduced or eliminated air emissions
- reduced or eliminated pollution discharges to water
- other, specify
- none
- don't know.
Thank you for your input!
Hans Scheifele
Office of Compliance
EPA Headquarters
-------
Appendix C Page C-7
OMB control number 2020-0015 Expires 3/31/08
Public reporting burden for this collection is estimated to average 3 minutes per response, including time
for reviewing instructions, gathering information, and completing and reviewing the information. Send
comments on the Agency's need for this information, the accuracy of the provided burden estimate, and
any suggestions for reducing the burden, including the use of automated collection techniques, to the
Director, Office of Environmental Information, Collection Strategies Division, United States
Environmental Protection Agency (mail Code 2822), 1200 Pennsylvania Ave, NW Washington, D.C.
20460; and to the Office of Information & Regulatory Affairs, Office of Management & Budget, 725 17th
Street, NW, Washington D.C. 20503, Attention: Desk Officer for EPA. Include the EPA ICR 1860.03
and the OMB control number 2020-0015 in any correspondence. Do not send your completed survey to
this address. Approval expires March 31, 2008.
-------
Page C-8 Appendix C
Frequently Asked Questions
Question 1: When do I need to use the generic compliance assistance ICR?
The generic clearance is to be used for voluntary collections of program evaluation information when
EPA wants to survey more than nine recipients of a compliance assistance activity. This generic ICR
can be used for the bulk of the planned program evaluation/outcome measurement work planned in the
regions and headquarters, where your goal is to determine the effectiveness of your compliance assistance
activities on the audience that receives the compliance assistance (e.g., participants at a workshop).
Programs are encouraged to use good survey design, methodology, and implementation in all of their
program evaluation work.
Note, however, that the generic ICR cannot be used when you plan to use a statistical approach to
generalize the effectiveness of a compliance activity on an overall population. In this case, you will have
to develop a separate ICR for your evaluation.
Question 2: Is an ICR required for all surveys? An ICR is not required for surveys or pre/post tests
that are given during a workshop that has been well publicized - open to the public.
Question 3: How do I receive approval for my survey, if it meets the conditions outlined above?
Prior to initiating the survey, sponsoring regional programs must seek final survey approval from OMB.
To obtain approval, the sponsoring regional program must submit a clearance package (a full example is
on pages 2-4) consisting of a memorandum and a copy of the survey instrument through CASPD first
who will forward the survey to the Collection Strategies Division (CSD) in the Office of Environmental
Information.
CASPD staff will review each submission to ensure that it meets the requirements of the Paperwork
Reduction Act and the generic approval, and may reject any proposed survey that does not meet those
requirements.
Question 4: How Long Will The Process Take?
Approximately 45 days at the most and 14 days at the least - this is a combination of EPA and OMB time.
EPA process: Send your package to the staff lead for the CA ICR in HQ/OC/CASPD. They will do a
quick review (1-2 days) and send it on to OECA's desk officer in OEI who will submit the survey and
attached materials to OMB.
OMB process: OMB will review surveys that evaluate satisfaction with or improved understanding that
results from compliance assistance within 10 working days. OMB will review surveys that evaluate
behavioral responses to compliance assistance (i.e., asking one or more of the five questions allowed to
evaluate behavioral change) within 30 working days.
Question 5: What are the requirements in the ICR valid through 2008? This generic ICR restricts the
behavior change questions that EPA can ask. You can only ask the following questions:
-------
Appendix C Page C-9
1. What regulatory actions do you intend to take (did take) as a result of the ?
a. File a notification
b. Obtain a permit or certification
c. Provide data to EPA, state, or local regulator
d. Submit reports to EPA, state, or local regulator
e. Comply with an environmental requirement
f. No regulatory actions will be taken
g. Other, specify
2. What process changes at your facility do you intend to make (did make) as a result of the
?
a. Change the storage or handling of a waste or emission
b. Change a process or practice
c. Purchase new process equipment
d. Implement material or waste recycling system
e. Install pollution control equipment (e.g., scrubbers, control technique)
f. Install a waste treatment system
g. Switch to renewable energy
h. No process changes will be taken
i. Other, specify
3. What management changes at your facility do you intend to take (did take) as a result of the
?
a. Conduct a self audit
b. Institute an environmental management policy, system or procedure
c. Institute training or other communication to improve awareness and/or practices
d. Identify pollution prevention opportunity
e. No management action will be taken
f. Other, specify
4. Who (if anyone) have you contacted (will contact) for further assistance as a result of the
?
a. A vendor
b. A state or local regulatory agency
c. A non-regulatory local source for additional compliance assistance
d. No contacts will be made
e. Other, specify
5. Please identify any reductions or the elimination of a pollutant(s) that resulted, or will result,
from the actions taken as a result of the CA activity.
-------
Page C-10 Appendix C
reduced or eliminated waste (solid/hazardous)
reduced or eliminated air emissions (air, fugitive)
reduced or eliminated pollutant emissions (wastewater)
other, specify
none
don't know.
6. Have you realized, or expect to realize, a cost savings from actions taken as a result of the
_Yes
No
-------
Appendix C PageC-11
Question 6: Can I Send a List of Survey Questions That I Plan to Use over and over Again at
Workshops, Training Seminars, or On-site observations to OMB for Approval?
Yes. If you are going to be repeating your compliance assistance activities over the course of the year(s)
you can send a comprehensive list of survey questions for pre-approval. The list of questions would be
broader than the list for any one survey. For example, you may know that you will be conducting MACT
training seminars for wood finishers, drycleaners and paint coaters next year. You also know that you
want to follow-up on the seminars by asking those that attended what changes they made at the facility as
a result of the training. You would develop your list of questions that applied across all three sectors plus
those specific to each sector for OMB approval. When you delivered the surveys you would only use
those questions that were appropriate to the audience being assisted. In calculating burden, however, you
will need to base it on the overall number of respondents to all of the surveys (see BURDEN calculation
example below).
Question 7: What Do I Have to Do to My Survey to Show That it Has Been Approved by OMB?
The OMB Control Number and expiration date must appear on the front page of an OMB-approved form
or survey, or on the first screen viewed by the respondent for an on-line application. The rest of the
burden statement must be included somewhere on the form, questionnaire, or other data collection tool, or
in the instructions or cover letter for such collection.
The following information must appear on the top of every page of the survey:
OMB Control No. 2020-0015 Approval expires March 2008
The following statement must appear somewhere in the actual survey, usually at the end:
Public reporting for this collection of information is estimated to average XX minutes per response,
including time for reviewing instructions, gather information, and completing and reviewing the
information. Send comments on the Agency's need for this information, the accuracy of the provided
burden estimate, and any suggestions reducing the burden, including the use of automated collection
techniques to the Director, Office of Environmental Information, 1200 Pennsylvania Ave., NW
Washington DC 20460 and to the Office of Information and Regulatory Affairs, Office of Management
and Budget, 725 17th St. NW Washington DC 20503. Attention: Desk Officer for EPA. Include the EPA
ICR 1860.03 and the OMB control number 2020-0015 in any correspondence. Do not send your
completed surveys to this address.
Question 8: Is There a Standard Cover Memo for the Clearance Package and, How Do I Calculate
the Necessary Burden Estimates?
Yes, a sample memorandum is on page 2. In the memo you need to address five areas:
1) Background: Briefly describe the compliance assistance activity being undertaken;
2) Survey Purpose and Description. Briefly describe the parameters that you are attempting to measure
through the survey and attach the actual survey (or list of survey questions);
3) Survey Methodology and Use of Results: Explain how you plan to conduct the survey (e.g., mail or
phone survey, on-site visit) and how you will use the results;
Question 9: Are there any caveats that should be made when publicly reporting on the results of
your survey approved under the generic ICR for compliance assistance?
Yes, you will need to include the following disclaimer language whenever you publicly report the results
-------
Page C-12 Appendix C
of your survey approved under the generic ICR for compliance assistance.
"These performance measures are not calculated from a representative sample of the regulated entity
universe. The percentages are based on the number of regulated entities that answer affirmatively to these
questions on our voluntary surveys. The percentages do not account for the number of regulated entities
who chose not answer these questions or the majority of entities who chose not to answer the survey."
4) Respondents Burden: Calculate total respondents' burden hours and total respondents' cost. Follow
the approach below and see the example in Attachment 1.
a) Number of Respondents: How many respondents do you anticipate responding to the survey.
b) Minutes per response: How long will it take to complete the survey. In the ICR, our estimates per
respondent were as follows: phone surveys-10 minutes; mailed/faxed back surveys-10 minutes;
pre/post tests-10 minutes; on-site revisits-120 minutes; and online surveys- 5 minutes.
c) Cost per hour: In the ICR, we estimated the cost per respondent at $39.40 plus a 110 percent
overhead for a total hourly rate of $82.74. This rate applies to both the regulated community and state
and local technical assistance staff. These rates are from the United States Department of Labor,
Bureau of Labor Statistics, September 2004, "Table 2: Civilian Workers, by occupational and
industry group." These rates are from column 1 "Total compensation ." The rates have been
increased by 110% to account for the benefits packages available to those employed by private
industry. You should use this figure in calculating respondents' cost.
Total Burden Hours = (# of Respondents X Minutes per response)/60
Total Respondents Cost = $82.74 X Total Burden Hours
5) Agency Burden: This refers to the time that it will take you to review the responses and conduct your
analysis. You will need to supply a Total Agency Burden estimate, which is calculated by multiplying
EPA staff time (in hours) by cost per hour. In the ICR, we estimated cost per hour as $49.29. This rate is
based on the 2007 GS-13-01 annual salary of $66,951 or $37.91/hr, plus a 30 percent overhead. This
figure should be used in your calculation of Agency Burden.
If you supply OMB with a list of survey questions that you will use throughout your regional compliance
assistance program, you will still need to perform the above calculations. However, the estimate should
reflect the total anticipated burden for all of the surveys that use questions from the list.
-------
APPENDIX D
June 2002
-------
Appendix D
Page D-l
MENU OF SAMPLE SURVEY QUESTIONS BY OUTCOME MEASURE
This appendix provides a menu of sample survey questions you can tailor for your specific evaluation
efforts. Use them as a guide to get you started. Add questions, as appropriate, and delete questions that
are unrelated to your evaluation effort. Modify the questions to fit the sector you targeted as well as for
the type of compliance assistance activity. If your survey is conducted under the generic ICR, questions
indicating behavioral and environmental changes must be limited to the five listed in Section I, parts B
and C below. Additional survey questions can be found at:
http://www.epa/gov/compliance/planning/results/surveys/index.html
Section I of this document focuses on outcome measurement, which is central to OECA's commitments
under the Government Performance and Results Act. Section II lists supplemental questions for on-site
visits and Web sites. Section III of this document includes sample questions to assess the background of
the respondent and customer satisfaction.
SECTION I:
OUTCOME MEASUREMENT
Measuring the outcomes of compliance assistance—changes in understanding, behavioral change, and
environmental and human heath impacts—is a central component of OECA's performance measurement
strategy. Table 1 lists the outcome measures.
Table 3-1. Outcome Measures for Compliance Assistance
Outcome
Measure
Changes in
Understanding
Behavioral
Changes
(also referred
to as
"Improved
Environmental
Management
Practices")
Environmental
and Human
Health
Improvements
Some Specific Measures
of Outcomes Resulting from Compliance Assistance
Number/percentage of facilities/respondents who say they better understand how to
comply with environmental regulations
Number of facilities that contacted someone for further compliance assistance
Number of facilities that took at least one recommended action to comply with
environmental regulations (e.g., labeling, recordkeeping, reporting, obtaining a
permit)
Number of facilities/respondents that adopted at least one process change (e.g.,
pollution prevention changes, best management practices)
Number of facilities improving environmental management systems or conducting
reviews (e.g., training, self-audits)
Compliance rate changes
Number of facilities that have changed regulatory status
Number of facilities that eliminated, treated or reduced emissions or other
pollutants
Quantified environmental improvements (e.g., amount of emissions or pollutants
eliminated, treated or reduced)
-------
Page D-2 Appendix D
A. Outcome Measurement Category: Understanding
1. Would you say that you are more knowledgeable about environmental requirements and opportunities
as a result of this compliance assistance?
Q Yes
a NO
a N/A
2. As a result of the assistance you received, how has your understanding of the environmental regulations
that apply to your business improved?
Q A great deal. I feel that I understand what is required.
Q Somewhat. I am still a bit confused about the regulations.
Q Not at all.
a N/A
Comments:_
3.What would have helped you to understand the environmental regulations more fully?
Q More clearly written regulations.
Q Better written guidance materials.
Q A more knowledgeable staff person.
Q A training class or workshop.
Q More time to read the materials.
Q Other:
4.What did you learn that will be most useful to you?
Q How to apply for a permit.
Q Information on new equipment or techniques to use to lower emissions.
Q How to implement an environmental management system.
Q The name of a contact in another regulatory department.
Q Information on how similar companies have reduced emissions or improved
compliance.
Q Other:.
B. Outcome Measurement Category: Behavioral Change
If your survey is conducted under the generic ICR, questions indicating behavior change are limited to the
four below:
l.What regulatory actions do you intend to take (did take) as a result of the ?
Check all that apply:
Q File a notification.
Q Obtain a permit or certification.
Q Provide data to EPA, state, or local regulator.
Q Submit reports to EPA, state, or local regulator.
Q No regulatory actions will be taken.
-------
Appendix D Page D-3
Q Other specify:
2.What process changes at your facility do you intend to make (did make) as a result of the ?56
Check all that apply:
Q Change the storage or handling of a waste or emission.
Q Change a process or practice.
Q Purchase new process equipment.
Q Implement material or waste recycling system.
Q Install pollution control equipment (e.g., scrubbers, control technique).
Q Install a waste treatment system.
Q No process changes will be taken.
Q Other specify:.
3. What management changes at your facility do you intend to take (did take) as a result of the
?
Check all that apply:
Q Conduct a self audit.
Q Institute an environmental management policy, system or procedure.
Q Institute training or other communication to improve awareness and/or practices.
Q No management action will be taken
Q Other specify:
4. Who (if anyone) have you contacted (will contact) someone for further assistance as a result of the
?
Check all that apply
Q A vendor.
Q A state or local regulatory agency.
Q A non-regulatory local source for additional compliance assistance.
Q No contacts will be made.
Q Other specify:.
C. Outcome Measurement Category: Environmental and Human Health Improvements
1. Of the activities that you have implemented as a result of the what
have been the resultant pollutant reductions and cost savings, if applicable?
Activity Pollution/Reduction Cost Savings
56Process changes resulting from a SDWA sanitary survey should be tracked in ICIS as both improved
environmental management practices and reduced pollution as noted in Sanitary Surveys and Other On-site Drinking
Water Compliance Assistance Reported into the Integrated Compliance Information System in FY 06, Transmittal
Memorandum, James R. Edward and Stephen Heare (July 30, 2007).
-------
Page D-4 Appendix D
SECTION II: SUPPLEMENTAL QUESTIONS FOR ON-SITE WORKSHOPS AND WEB SITES
A. Supplemental Questions for Workshops
1. Was the material presented clearly and in a logical sequence?
Q Yes
a NO
2. How would you rate the handouts and materials?
Q Excellent
Q Very Good
Q Good
a Fair
Q Poor
3. Would you like more seminars like this one made available to you?
Q Yes
a NO
If yes, please list topics:
4. What was the most useful part(s) of the workshop?
5. What was the least useful part(s) of the workshop?
6. What topic(s) would you have liked to have spent more time on:
7. Would you be willing to spend more than 2 hours at a workshop in order to cover more topics?
Q Yes
a NO
8. Do you have any suggestions to help us reach more people like yourself to have them attend these
seminars?
9. Was the location of this meeting convenient for you?
-------
Appendix D Page D-5
Q Yes
a NO
Q Somewhat
Comments:
10. What other locations would you recommend for future seminars?
11. Was the time of the seminar convenient for you?
Q Yes
a NO
Q Somewhat
12. What other times might be more convenient?
Q Morning (8:00 a.m. - 11:00 a.m.)
Q Lunch (12 noon - 2:00 p.m.)
Q Afternoon (2:00 p.m.-5:00 p.m.)
a Evening (7:00 p.m.-9:00 p.m.)
13. Did the speakers show knowledge of the subject?
Q Yes
a NO
14. Was the technical level right for you?
Q Yes
a NO
15. Were the questions handled appropriately?
Q Yes
a NO
16. What did you learn that will be the most helpful to you?
17. On a scale of 1-10(10 being the highest) how would you rate:
The workshop
The presenters
18. Would you be interested in having a followup on-site compliance assessment?
Q Yes
a NO
B. Supplemental Question for Web Sites
1. How did you learn about this Web site?
Q Search engine (please specify):
Link from another Web site (please specify):
-------
Page D-6 Appendix D
Q From an EPA document
Q Referral from a colleague
Q Other (please specify):
-------
Appendix D Page D-7
SECTION III: SUPPLEMENTAL BACKGROUND AND CUSTOMER SATISFACTION QUESTIONS
A. Supplemental Background Questions
1. What type of organization do you work for?
Q Regulated facility or business
Industry sector:
Q Consulting company or law firm
Q Government
Q Trade association
Q Nonprofit organization
Q School or university
2. How did you become aware of this [insert name of compliance assistance activity]?
Q Referral from another government agency, official, or hotline
Q Referral from another business
Q Trade association
Q EPA letter or mailing
Q EPA workshop, seminar, or conference
Q Web site
Q EPA publication or newsletter
3. In the past, have you used any other compliance assistance tools provided by EPA, such as (check all
that apply):
Q Hotline
Q Fact sheets
Q Guidance documents
Q Web site
Q On-site visits
Q Workshops, seminars, or conferences
-------
Page D-8 Appendix D
4. In what areas did you request assistance? (Check all that apply)
Q Air permitting or regulations
Q Water permitting or regulations
Q RCRA/Hazardous waste permitting or safe handling of waste
Q Community right-to-know regulations
Q Toxic substances
Q Pesticides
Q Underground storage tanks
5. What prompted you to seek assistance? (Check all that apply.)
Q To find out if a specific environmental regulation applies to my facility.
Q General Information about regulations.
Q Need help filling out a permit application form.
Q To identify ways to change status from regulated to unregulated business.
Q To obtain Information about equipment or processes that will help us save money
complying with regulations.
Q To learn about pollution prevention opportunities.
Q Other:
6. Did the assistance provided adequately address the need you identified in Question 5 above?
Q Yes
a NO
B. Supplemental Customer Satisfaction Questions
The first four questions are "core " customer satisfaction questions from EPA 's customer
feedback and customer satisfaction measurement guidelines. When using these questions alone, EPA
recommends using a scale ofl to 6 for customer satisfaction surveys for consistency. When combining
customer satisfaction questions with questions to measure outcomes, you might want to adjust the scale
for consistency. While a range of answers is good for customer satisfaction surveys, this approach makes
less sense for outcome measurement as EPA needs precise answers to track outcomes.
1. Overall, how satisfied are you with the services and products you have received from EPA?
123456
not at all very much
-------
Appendix D
2.
3.
4.
How courteously did EPA staff treat you?
12345
not at all
How satisfied are you with the communications you have received from EPA?
12345
not at all
How fully did EPA respond to your needs?
12345
not at all
Page D-9
6
very much
6
very much
6
very much
5. Would you recommend this [insert name of compliance assistance activity] to other businesses?
a
a
Yes
No
6. How would you rate the technical understanding of the person assisting you (for helplines,
workshops, on-site assistance) or the technical quality of materials (for publications, Web sites, and
other materials)?
a
a
a
a
a
Excellent
Good
Fair
Poor
No way to tell
7. How can we improve delivery of this service?
-------
APPENDIX E
-------
Appendix E Page E-l
Examples of Letters Used by OECA to Gather Data by the Dillman Method
Example Metal Finishing Prenotification Letter for Survey Recipients
Dear Metal Finisher:
Over the past several years, EPA has conducted several compliance assistance efforts targeted to metal finishing facilities to improve
environmental quality and compliance rates. EPA has recently begun to measure the results of these efforts and to identify whether
additional compliance assistance is needed.
To support these efforts, EPA will be conducting a study of metal finishers in EPA Regions 1 and 557 to identify areas where additional
compliance assistance is needed, to identify the most useful types of compliance assistance, and to develop a "snapshot" of current
performance towards key Federal environmental regulations. EPA will use both mailed surveys and site visits to conduct this study in
order to compare the efficiency and effectiveness of these two data gathering techniques. This study will be a voluntary blind study.
This means that EPA will NOT know the identity of the facilities participating in the study. This study is NOT intended to provide
information on specific facilities, but instead is intended to develop aggregate statistics on a subset of the industry to evaluate
compliance assistance needs. There will NOT be any enforcement follow-up activities related to this study.
EPA has contracted with Eastern Research Group, Inc. (ERG), to conduct the study. ERG has randomly selected 100 metal finishers in
EPA Regions 1 and 5 for site visits, and has randomly selected another 200 facilities to receive the mailed survey. This letter is a
notification that your facility has been randomly selected to receive a mailed survey. Your participation is voluntary; however, by
participating you will provide EPA with information on how EPA can improve compliance assistance delivered to metal finishers, and
you will help EPA measure the effectiveness of its compliance assistance projects. As stated above, this is a blind study; EPA will not
know which facilities participate in the study. All surveys will be identified by code; facility names or addresses will not be included on
the surveys. ERG randomly selected the facilities to participate in this study from an industry population identified through three data
sources: the Harris Directory, EPA Regions 1 and 5, and Publicly Owned Treatment Works.
Within the next one to two weeks, you will receive a mailed survey developed by EPA to evaluate awareness of and performance
towards the following key Federal environmental regulations: the solvent degreasing NESHAP standard, the chrome MACT standard,
Clean Water Act requirements, RCRA requirements, and EPCRA reporting requirements. The survey also includes several questions to
assess what types of compliance assistance have been used by the facility, and what types of compliance assistance facilities find most
useful.
ERG has established a helpline to support any facilities needing additional information or guidance on completing the checklist. The
helpline number is 1-866-867-4637. In addition, EPA will provide a self-addressed stamped envelope for returning the survey. To help
maintain the confidentiality of the responses, please do not add a return address to this envelope.
To maintain our project schedule, EPA will be requesting that you complete and return the survey within two weeks of receipt. The
results of this study will be published on EPA's Web site and will also be made available to the Association of Electroplaters and
Surface Finishers (AESF). If you have any questions or concerns, please call the helpline.
EPA recognizes that you have many demands on your time. This study will help EPA develop and deliver better focused compliance
assistance tools that we hope will save you time and money in the future. We thank you in advance for participating in this study.
Sincerely,
Linda Darveau
Environmental Specialist
US EPA New England
(617) xxx-xxxx
57EPA Region 1 comprises of Connecticut, Maine, Massachusetts, New Hampshire, Rhode Island, and Vermont,
while EPA Region 5 comprises of Illinois, Indiana, Michigan, Minnesota, Ohio, and Wisconsin.
-------
Page E-2 Appendix E
Example Metal Finishing Prenotification Letter for Site Visits Recipients
Dear Metal Finisher:
Over the past several years, EPA has conducted several compliance assistance efforts targeted to metal finishing facilities to improve
environmental quality and compliance rates. EPA has recently begun to measure the results of these efforts and to identify whether additional
compliance assistance is needed.
To support these efforts, EPA will be conducting a study of metal finishers in EPA Regions 1 and 558 to identify areas where additional
compliance assistance is needed, to identify the most useful types of compliance assistance, and to develop a "snapshot" of current performance
towards key Federal environmental regulations. EPA will use both mailed surveys and site visits to conduct this study in order to compare the
efficiency and effectiveness of these two data gathering techniques. This study will be a voluntary blind study. This means that EPA will NOT
know the identity of the facilities participating in the study. This study is NOT intended to provide information on specific facilities, but
instead is intended to develop aggregate statistics on a subset of the industry to evaluate compliance assistance needs. There will NOT be any
enforcement follow-up activities related to this study.
EPA has contracted with Eastern Research Group, Inc. (ERG), to conduct the study. ERG has randomly selected 100 metal finishers in EPA
Regions 1 and 5 for site visits, and has randomly selected another 200 facilities to receive the mailed survey. This letter is a notification that your
facility has been randomly selected for a site visit. Your participation is voluntary; however, by participating you will provide EPA with
information on how EPA can improve compliance assistance delivered to metal finishers, and you will help EPA measure the effectiveness of its
compliance assistance projects.
As stated above, this is a blind study; EPA will not know which facilities are visited. All site visit information will be identified by code; facility
names or addresses will not be included in any reports. ERG randomly selected the facilities to participate in this study from an industry
population identified through three data sources: the Harris Directory, EPA Regions 1 and 5, and Publicly Owned Treatment Works.
Within the next week, you will receive a phone call from ERG to determine the best available date and time to schedule the site visit. During the
2-4 hour site visit, an ERG employee will complete a copy of the survey developed by EPA to evaluate awareness of and performance towards
the following key Federal environmental regulations: the solvent degreasing NESHAP standard, the chrome MACT standard, Clean Water Act
requirements, RCRA requirements, and EPCRA reporting requirements. The survey also includes several questions to assess what types of
compliance assistance have been used by the facility, and what types of compliance assistance facilities find most useful. The results of this study
will be published on EPA's Web site and will also be made available to the Association of Electroplaters and Surface Finishers (AESF).
EPA recognizes that you have many demands on your time. This study will help EPA develop and deliver better focused compliance assistance
tools that we hope will save you time and money in the future. We thank you in advance for participating in this study.
Sincerely,
Uylaine Barringer
Assistant Program Manager
RCRA Compliance.
58EPA Region 1 comprises Connecticut, Maine, Massachusetts, New Hampshire, Rhode Island, and Vermont, while
EPA Region 5 comprises Illinois, Indiana, Michigan, Minnesota, Ohio, and Wisconsin.
-------
Appendix E Page E-3
Example Metal Finishing Cover Letter for Survey Recipients
Dear Metal Finisher:
Within the past two weeks, you should have received a letter notifying you of an EPA study of metal finishers to identify
areas where additional assistance is needed related to compliance with environmental regulations, to identify the most
useful types of compliance assistance, and to develop a "snapshot" of current performance towards key Federal
environmental regulations. This study is a voluntary blind study. This means that EPA will NOT know the identity of
the facilities participating in the study. This study is NOT intended to provide information on specific facilities, but
instead is intended to develop aggregate statistics on a subset of the industry to evaluate compliance assistance needs.
There will NOT be any enforcement follow-up activities related to this study. Your facility has been randomly selected
to participate in this study. Your participation is voluntary; however, by participating you will provide EPA with
information on how EPA can improve compliance assistance delivered to metal finishers, and you will help EPA
measure the effectiveness of its compliance assistance projects.
HOW DO YOU PARTICIPATE?
To support EPA in this effort, we ask that you:
1) Complete the attached survey. The survey is designed to evaluate awareness of and performance towards key Federal
environmental regulations, and to assess what types of compliance assistance you have found useful.
2) Call the survey toll-free helpline number at 1-866-867-4637 if you have any questions on how to complete the survey.
3) Place the survey in the enclosed self-addressed stamped envelope and return it by two weeks after receipt.
EPA recognizes that you have many demands on your time. This study will help EPA develop and deliver better focused
compliance assistance tools that we hope will save you time and money in the future. We thank you in advance for your
time in participating in this study.
Sincerely,
Linda Darveau
Environmental Specialist
US EPA New England
(617)xxx-xxxx
-------
Page E-4 Appendix E
Example Metal Finishing Prenotification Letter for Site Visits Recipients
The following text appeared on a 5.5" x 8" postcard made from blue cardstock.
Metal Finishing Performance
Survey Reminder
Dear Metal Finisher,
Please remember to complete the EPA Metal Finishing Performance Evaluation Survey and return it in
the enclosed, self-addressed stamped envelope by December 14th. If you have any questions regarding
survey completion, please contact the toll-free helpline at 1-866-867-4637.
If you have already returned your completed survey, thank you for your participation.
Sincerely,
Linda Darveau
US EPA New England
-------
Appendix E Page E-5
Example Metal Finishing Re-mail Cover Letter for Survey Recipients
Dear Metal Finisher:
In late October, you should have received a survey regarding an EPA study of metal finishers to identify areas where additional
assistance is needed related to compliance with environmental regulations, to identify the most useful types of compliance assistance,
and to develop a "snapshot" of current performance towards key Federal environmental regulations. This study is a voluntary blind
study. This means that EPA will NOT know the identity of the facilities participating in the study. This study is NOT intended to
provide information on specific facilities, but instead is intended to develop aggregate statistics on a subset of the industry to evaluate
compliance assistance needs. There will NOT be any enforcement follow-up activities related to this study. We have not yet received
your survey. Your participation is voluntary; however, by participating you will provide EPA with information on how EPA can improve
compliance assistance delivered to metal finishers, and you will help EPA measure the effectiveness of its compliance assistance
projects.
HOW DO YOU PARTICIPATE?
To support EPA in this effort, we ask that you:
1) Complete the attached survey. The survey is designed to evaluate awareness of and performance towards key Federal environmental
regulations, and to assess what types of compliance assistance you have found useful.
2) Call the survey toll-free helpline number at 1-866-867-4637 if you have any questions on how to complete the survey.
3) Place the survey in the enclosed self-addressed stamped envelope and return it by two weeks after receipt.
EPA recognizes that you have many demands on your time. This study will help EPA develop and deliver better focused compliance
assistance tools that we hope will save you time and money in the future. We thank you in advance for your time in participating in this
study.
Sincerely,
Linda Darveau
Environmental Specialist
US EPA New England
(617) xxx-xxxx
-------
APPENDIX F
-------
Appendix F Page F-l
Advanced Data Analysis -Measuring Association and Making Comparisons
The descriptive statistics discussed in Section V above were only concerned with measuring
characteristics of one variable at a time. In this section we discuss measure of association and how to
make statistical comparisons (i.e., hypothesis testing) when conducting data analysis. We expect that each
of the methods discussed in this section will be accessible to individuals with basic math skills. We
describe some more advanced techniques below. The more advanced techniques may require additional
support from a qualified statistician.
Measures of Association
Measures of association deal with how two or more variables relate to one another. There are two
measures of association that you should be aware of: correlation coefficients and contingency tables.
Correlation coefficients measure the direction and strength of the relationship between two variables.
They are most useful when both of the variables you are concerned with can take on unlimited values.59
Due to complicated nature of the calculation, we exclude the formula for calculating correlation. Most
spreadsheets, however, have a correlation function that is relatively easy to apply. The following can
assist you in interpreting correlation coefficients:
1. A correlation coefficient can take on values between -1.0 and 1.0.
2. Negative values of the correlation coefficient indicate an inverse relationship between the two
variables: i.e., as one variable is increasing in value, the other is decreasing in value.
3. Positive values of the correlation coefficient indicate a positive relationship between the two
variables: i.e., as the value of one variable increase, the value of the other variable also
increases.
4. Correlation coefficients that are close to (or equal to) zero indicate no correlation between the
two variables.
5. Larger values of the correlation coefficient (in absolute terms) indicate stronger correlations.
For example, a correlation of 0.65 indicates a stronger relationship than a correlation of 0.35.60
A correlation of 1.0 or -1.0 should be interpreted with caution. It is rare to find to variables that are
perfectly correlated with one another. If this occurs, you should make sure that one variable is not being
derived from the other.61 Another caution you should consider is that correlation does not imply
causation. That is, a positive correlation indicates that when one variable takes on a large value, so does
the other variable. It does not imply that the large value of one variable caused the other variable to also
be large.
59 See Section V's definition of limited variables.
60 The same is true for negative correlations, a -0.65 is a stronger correlation than a -0.35.
61 For example, if we had information on the pounds of a chemical used by a company and the total cost of that
chemical, we should find a perfect positive correlation if we use the same dollar amount for each observation to
derive the cost.
-------
Page F-2
Appendix F
When you have three or more variables that you are interested in, a "correlation matrix" is a useful
means of displaying the correlation coefficients. To form such a matrix, simply write the variable names
across the column headings and then (in the same order) down the row headings. Then, fill in the
correlation coefficients for each pairing below the diagonal.62 The following table provides an example of
what a four-variable correlation matrix would look like.
Variables
A
B
C
D
A
1.0
0.85
0.65
0.01
B
-
1.0
0.75
0.02
C
-
-
1.0
-0.25
D
-
-
-
1.0
The above-diagonal cells can be left blank since they are redundant with the below-diagonal cells. Note
that in the example, variables A, B, and C are all strongly positively correlated with one another. Variable
D, on the other hand, is not correlated with A and B and only weakly and negatively correlated with C.
The second form of measuring association are contingency tables, or cross-tabulations. Contingency
tables are useful when you are interested in looking at discrete or limited-value variables. With a
contingency table you can look at the relationship between the values for two or more variables. There are
no set ways of developing contingency tables, so we provide a few examples. The key, however, is to
form a set of columns from the answers to one or more questions and form rows from the answers to one
or more different questions.
Suppose you are interested in how small and large entities differed in their answers to a variety of
survey questions regarding the usefulness of certain compliance assistance tools. The following table
could help you look at that question, taking into account that not all entities accessed all the tools.
Tools:
Facility
Size
Small
(305 total)
Large
(151 total)
TOTAL
Tool A
Used Tool
305
151
456
Found it
Useful
102
34
136
ToolB
Used Tool
305
62
367
Found it
Useful
98
57
155
ToolC
Used Tool
202
47
249
Found it
Useful
198
44
242
This table combines information on how many accessed each tool, the size of those that accessed each
tool, and how many (of both size classes) that found each tool useful. As can be seen in the table, Tool A
was not very useful but Tool C was useful to respondents. The usefulness of Tool B differed between
1 The diagonal of the matrix is each variable correlated with itself, which by definition should equal one.
-------
Appendix F
small and large facilities with large facilities finding it more useful than small facilities.
Page F-3
As another example, suppose you are interested in the relationship between the answers to two survey
questions: one that asked respondents to rate their satisfaction with a compliance assistance program on a
scale of one to five (five = very satisfied) and another that asked respondents to identify which
compliance assistance tools they had used (Tools A, B, and C, as above). A contingency table for this
question might look like this:
Satisfaction With Program
Level
1
2
3
4
5
TOTAL
Total Responses
56
79
159
125
37
456
Tools Accessed By Those Respondents
A
56
79
159
125
37
456
B
34
65
131
102
35
367
C
5
19
75
118
32
249
In this example, we can see a clear pattern: satisfaction with the program increased with use of Tool C
and to a lesser degree with Tool B. No additional information on the relationship between use of Tool A
and satisfaction is possible, however, since all respondents accessed Tool A.
Making Comparisons (Hypothesis Testing)
Hypothesis testing is the backbone of inferential statistics and as such is one the most important aspects
of statistical analysis. In its most basic form, hypothesis testing attempts to determine if an observed value
or relationship is attributable to random chance. Values or relationships that are not attributable to random
chance are generally referred to as statistically significant. In this discussion we provide some basic
information on when hypothesis testing should be employed and we also provide information on a few
simple tests that you can use. For the most part, complicated hypothesis tests should be left to those with
good statistical backgrounds.
Hypothesis tests should be employed when you want to verify that a certain value or relationship is not
due to simple random chance. For example, you may want to test whether the compliance rate among
respondents was significantly greater than 50 percent. You may want to determine whether small and
large entities had the same rate of satisfaction with a compliance assistance program. As a final example,
you may want to determine whether a calculated correlations coefficient was significantly different from
zero. In other words, you will want to conduct hypothesis tests if you want to make inferences
(statements) about your data (or the underlying population) with some confidence.
To simplify our discussion of hypothesis testing, we focus on four tests that should be most useful to
you:
• One Sample Test for a Proportion—A test to determine if a percentage is significantly different
than a specific value.
Two Sample Test for a Proportion—A test to determine if a percentage calculated from one
-------
Page F-4 Appendix F
sample is significantly different than a percentage calculated from another sample.
• One Sample Test for a Mean—A test to determine if an average value is significantly different
than a specific value.
• Two Sample Test for a Mean—A test to determine if an average calculated from one sample is
significantly different than an average calculated from another sample.
As can be seen, the four tests cover one and two samples and proportions and averages. In a one sample
test, we are testing to see if the value in the sample is significantly different than a specific value. In a
two-sample test, we have values calculated from two different samples and we are looking a significant
difference between the two values. In the two-sample test, the two samples can be sub-groups from one
larger sample (e.g., larger versus small facilities). A proportion is any variable whose "mean" value is
expressed as a percentage (e.g., the percentage of facilities that responded with a "yes" to certain
question). Mean values are simply non-percentage values for the means of variables. We present the
formulas for these tests in Table F-l.
-------
Appendix F
Page F-5
Table F-l. Equations for Performing Simple Hypothesis Tests
Proportions
Means
One-Sample
\
Two-Sample
\
Si
Si
fe2
S2
Definitions
z-A statistical parameter used to test for significance.
ps-The estimated value of the proportion from the sample.
p-The value that you want to compare (statistically) ps to.
n-The number of observations in the sample.
Pb p2~The estimated proportion values for samples 1 and 2
respectively.
nb n2-The number of observations in samples 1 and 2 respectively.
z-A statistical parameter used to test for significance.
xs-The estimated mean value from the sample.
u-The value that you want to compare (statistically) xs to.
n-The number of observations in the sample.
xb x2-The estimated mean values for samples 1 and 2 respectively.
nb n2-The number of observations in samples 1 and 2 respectively.
Significance
Criteria
If the absolute value of z is:
-less than 1.65, then the difference is not statistically significant.
-between 1.65 and 1.96, then the difference is significant at the ten percent level of significance.
-between 1.96 and 2.58, then the difference is significant at the five percent level of significance.
-greater than 2.58, then the difference is significant at the one percent level of significance.
Note: The absolute value is calculated by simply turning a negative number positive or by making no adjustment to a positive number.
-------
Page F-6
Appendix F
In the table, we define a few "significance criteria" which tells you how to translate the values you
estimate into a level of significance. These criteria result in one of four levels of significance: not
significant, 10 percent, five percent, and one percent. The percent levels refer to a level of confidence in
the results. For example, the five percent level of significance implies that there is only a five percent
chance that your hypothesis test was "wrong". That is, there is some chance that you obtained a sample
that is not representative of the true population and thus your test has given you a false answer. To
interpret the results of your tests, you can use the following criteria defined in Table F-2.
Table F-2. Interpreting Hypothesis Test Results
Test
One Sample Test
for a Proportion
Two Sample Test
for a Proportion
One Sample Test
for a Mean
Two Sample Test
for a Mean
Interpretation
A significant difference (at any level) implies that your estimated sample proportion (ps) is
significantly different than the test value of p. The relative magnitudes of the two values
will tell you whether the sample proportion is significantly greater than or less than the
test value.
A significant difference (at any level) implies that the estimated proportion from the first
sample (p,) is significantly different than the estimated proportion from the second sample
(p2). The relative magnitudes of the two values will tell you which one is significantly
greater.
A significant difference (at any level) implies that your estimated sample mean (xs) is
significantly different than the test value of u. The relative magnitudes of the two values
will tell you whether the sample mean is significantly greater than or less than the test
value.
A significant difference (at any level) implies that the estimated mean from the first
sample (xj) is significantly different than the estimated mean from the second sample (x2).
The relative magnitudes of the two values will tell you which one is significantly greater.
Advanced Data Analysis-Additional Techniques to Analyze Data
In this section we introduce some more advanced techniques that can be used to look at relationship and
trends in your data. We present these techniques in only summary form since properly applying them will
require someone that has experience in applying these more advanced techniques. In our summaries, we
attempt to provide some indication of what each of these techniques can be used for and what they might
be able to provide you in terms of final results.
Analysis of Variance (ANOVA)—In an ANOVA analysis, the idea is to determine the influence that
different factors have had on some variable. Usually the factors of interest are qualitative in nature. For
example, suppose you have conducted some compliance assistance seminars to improve understanding
and you asked participants to take a pre- and post-seminar test. The seminar was conducted by three
different outside instructors and you conducted the seminar in two different ways. After conducting all
of the seminars, you want to look at the improvement in test scores (from the pre- to the post-seminar
test). There is one complication, however: you have three different instructors and two different ways in
which the seminar was conducted. An ANOVA analysis can assist you in sorting through the influence
of these different factors. In this case, an ANOVA will tell you which factors had the biggest influence
toward increasing scores between the pre- and post-seminar test.
Regression Analysis—The purpose of a regression analysis is to look at the influence of quantities of
different variables on one variable. Thus, while ANOVA looks the influence of different qualitative
factors, a regression analysis would look at the influence of quantitative factors on a specific variable.
-------
Appendix F Page F-7
For example, suppose you had information on the amount of chemicals used by a set of facilities, as well
as a number of other quantitative factors that you think may "explain" (i.e., influence) the variation in
chemical usage (e.g., capacity of final product, etc).63 A regression analysis would allow you to estimate
the quantitative relationship between each explanatory factor and chemical usage.
Analysis of Category Data—Categorical data analysis is an extension of the contingency tables
discussed above. This form of analysis is useful when the relationships you are interested in are
between variables that are categorical in nature (e.g., EPA Region, states, type of process, compliance
status, compliance assistance tools used, etc). For example, suppose you have information on the
compliance status of several entities and the types of compliance assistance tools each has used
(including none). Suppose you are interested in which tools are associated with higher levels of
compliance. A categorical data analysis would provide this type of information by performing a
hypothesis test to determine if compliance status is related to use of different tools. The analysis could
tell you which tools are associated with higher levels of compliance.
Drawing Appropriate Conclusions
In performing statistical analysis one must be careful to draw appropriate conclusions from the results.
In particular, there are three issues that you should consider when you are drawing conclusions.
First, you should be aware that statistical results that are based on a sample are subject to some degree
of error. That is, since the results are not based on the whole population, you are only generating
estimates which you then assume are representative of the population. Given this, there is still some
chance that you drew an unrepresentative sample and that your results are not accurate. This can happen
even if you follow all of the prescription in Section V on sampling. Thus, it is necessary to remember that
your results may be wrong. You should always report the size of your sample, the size of the population
that it is meant to represent, how you drew the sample or collected the data, and non-response rates for
surveys.
Second, statistical analyses are subject to the principle of "garbage in, garbage out." A sophisticated
statistical analysis will not overcome bad data. For example, you may hire a statistician to perform several
advanced analyses of survey data that you collected to answer a number of important questions. However,
if the survey data is flawed (e.g., a non-random sample, too few observations collected, rampant
nonresponse), then the resulting analyses will also be flawed. Thus, your analysis can only be as good as
your data.
Finally, statistical analyses do not prove causation. A statistical analysis can only demonstrate
tendencies or how two or more variable relate to one another. For example, suppose you are looking at
the relationship between a set of compliance assistance tools and improvements in compliance status, and
you find that one tool is strongly related to improved compliance. This does not prove that the use of the
tool caused an improvement in compliance. The strongest statement that you can make is that the tool is
strongly related to improved compliance. There may be other factors not accounted for in the analysis, or
the relationship between the tool and improved compliance may more complicated than a simple cause-
effect relationship. In other words, there may be other explanations for the observed relationship that the
statistical analysis cannot test for or encompass. Thus, it is necessary not to place an interpretation on
your results that the analytical framework cannot support.
63 Actually, not all of the factors need to be quantitative. Regression analysis can also encompass some qualitative
factors.
-------
APPENDIX G
-------
Appendix G Page G-l
Sampling From Unknown Populations
(Draft from 06/13/02)
The Challenge
One the most important components of any survey project, whether you are conducting on-site visits, a
paper survey, an Internet questionnaire or a phone survey, are the people who answer your survey. For
your survey to be valuable you need to reach a set of people who are representative of the overall group
that interests you. In statistical terminology, the set of people you contact is your sample and the whole
group is the population. You do not need to contact everyone in the population since you can obtain
reliable and accurate information from a sample while substantially reducing your data collection cost.
For example, lets say there are a total of 1,000 dry cleaners in your area but you're only going to survey
100. In this case, the 1,000 is the total population and 100 is the sample. Additionally, if you construct
your project correctly, the 100 dry-cleaners you survey should provide accurate and reliable information
on the characteristics and practices of the 1,000 dry cleaners in the population.
The difficulty arises when the population size is unknown or you have little information on who is in
the population. In this situation, you need to develop a reliable and accurate population list from which to
draw the sample of survey participants. This applies to both personal interview projects as well as written
surveys. Without a reliable and accurate list, your results may be skewed or biased. In these situations,
developing this list is one valuable outcome of the project.
Purpose of the Guidance
This guidance is intended to assist you in developing a reliable population list, from which to derive (1)
a random sample for statistically valid measurements or (2) an informal sample for non-statistically valid
measurements. A reliable list produces accurate information on as much of the intended population as
possible without including entities that are not part of the population.
We begin by discussing the characteristics of populations that are generally unknown or for which
reliable lists are typically not available (Section I). We then turn to potential data sources and ideas for
developing population lists (Section II). In our discussion of data sources, we try to give some insight into
how you develop reliable lists from multiple sources. Section III discusses how to estimate the population
size from the lists you have developed. It is important to estimate your population size before you take
measurements from that population. Section IV provides some simple ideas on how to draw samples from
these populations to obtain reliable and useful information.
I. Characteristics of Unknown Populations
Several characteristics are common among unknown populations. These characteristics stem from the
reasons why the businesses are not found on population lists. For sectors dominated by businesses that fit
these characteristics, definitive population lists are rare. The characteristics that we list below are certainly
not comprehensive, but should provide you with an idea of why some populations are unknown.
Sectors dominated by small businesses—Sectors with a majority of small businesses (e.g., "Mom and
Pop" operations, high attrition rate businesses, small annual sales) tend to have unknown populations.
These businesses have fewer resources, limiting their participation in trade organizations.
-------
Page G-2 Appendix G
Also, in a physical sense these businesses are less noticeable since they are smaller. Finally, groups of
small businesses may be less likely to form trade organizations than groups of larger businesses.
Less-formalized or informal businesses—There are a number of sectors where some (or most) of the
businesses operate in an informal manner. For example, the home remodeling sector contains a range
of business organizations from incorporated contractors to handyman services, and the auto salvage
sector includes many "hobbyists." It may be relatively easy to find lists of more formal business
operations (e.g., contractors), but less easy to find lists of less formal operations. In some cases, the less
formal operations may be actively seeking to avoid detection for tax purposes.
Businesses without extra resources—Businesses operating with extremely tight margins frequently do
not pay to participate in trade associations or other types of listing activities. Sectors where this is
common will thus tend to have unknown populations.
Situations where the population definition and business sectors do not match perfectly—For
example, you may be interested in chromium electroplaters, a form of metal finishing. Chromium
electroplaters, however, are not usually identified separately from other metal finishers. Thus, a list of
metal finishers that doesn't identify the type of electroplating operations may be too broad.
Sectors where there are language barriers—Sectors where the majority of businesses do not speak
English or where English is spoken as a second language may also tend be unknown populations. In
these cases, language works as a barrier to the development of population lists.
Seasonal or economy-driven sectors—There are numerous sectors where the season or the state of the
economy either determines their existence or heavily influences the number in the sector. Finding
reliable lists for these populations can be challenging since businesses within the sector come and go
frequently.
II. Data Sources
Table 1 lists several good data sources to develop population lists. In Table 2, we provide some
comments on other sources that may not be as useful64. You should consider the following as you develop
your population list:
Use more than one source—For unknown populations, you should use more than one source in
developing your population list. As you increase the number of sources, you are increasing the
reliability of your list. For one, coverage may differ between sources with some sources missing certain
groups of the population while others may contain those groups. Using multiple sources will result in
overlap; you should take note of overlapping entities.65 In using multiple sources, you should track the
sources for each entity that you identify.
Evaluate each source—In evaluating the usefulness and reliability of any source you need to
determine how the source developed its list, including the reliability of the data used. You need to know
64 The purpose of Table 2 is to steer you away from the sources in that table.
65 Overlapping can also be seen as a measure of reliability. Specifically, if you have two sources that you feel are
fairly accurate but somewhat incomplete, you can be fairly certain that the overlapping entities are part of the
population.
-------
Appendix G Page G-3
if inclusion in the list required some form of payment or fee.66 Finally, you need to know when the list
was last updated since older lists tend to be less accurate or complete. You should make notes of all of
this information to better assess the reliability of your final list.
Identify any biases that may result from the sources you use—An important aspect of developing
your population list is to recognize that the resulting list may be biased. For example, you may have
developed a population list for auto repair shops from a trade association list, the yellow pages, and a
marketing data service. Each of these may be biased in some way. For example, the trade association
may require annual dues, limiting the membership list to shops that can afford to be members, and not
everyone pays to be included in the yellow pages. Reviewing how the source list was developed will
allow you to determine what biases may exist in your final list. For example, all of your sources may be
biased against inclusion of smaller facilities. Using multiple sources should reduce the bias in your final
list.
Look at the total acquisition cost—A final consideration in developing your list is to consider the
trade-off between dollars and labor hours in developing population lists. Each source will require some
resources to acquire. Some sources can be obtained by simply purchasing the data from a company that
provides the information (e.g., Dun and Bradstreet, a company that specializes in collecting data on
public and private companies). Other sources may be free, but may require substantial labor effort (on
your part) to verify the reliability and accuracy of the data. In considering any potential source you
should consider what you are getting from the source (reliability and completeness) compared to what
you are paying for that source (dollars and/or labor hours).
66 For example, a trade association may publish a list of members, but may require a fee from the members for
inclusion in the list.
-------
Page G-4
Appendix G
Table 1. Some Potential Sources for Developing Population Lists
Source
Trade Associations
Phone Book
Federal or State
Agency Lists
Data Collection
Companies
Business Licensing
Departments
Comments
Numerous trade associations operate in the United States, many of which maintain comprehensive member lists. Although there are
several directories of trade associations, the more comprehensive ones are commercial products and it may not be cost-effective to
purchase one if all you need are names of a few trade associations. Many libraries have the trade association directories in print
form. One common and comprehensive directory is the National Trade and Professional Associations of the United States
(Columbia Books, published annually). Additionally, you can use the Internet to locate relevant trade associations. One
consideration for trade associations is to determine if all members of the association will be part of your target group.
The local-area phone book is a reliable source for locating businesses. CD-ROMS that contain Yellow Page listings are available
and offer flexible search options. You should determine the extent to which your targeted group fits into the categories offered in the
Yellow Pages.
A number of Federal and state agencies maintain lists of companies for various reasons. OSHA, for example, maintains a database
of all inspected companies (the Inspection Management Information System (IMIS)). One thing you should be aware of though is
that inclusion in these lists often depends on a number of criteria. For example, inclusion on OSHA's IMIS list requires an
inspection and that the company employ more than 10 people. (The OSHA IMIS database can be accessed at
http://www.osha.gov/oshstats/index.html.) In addition, state and local agencies may also have regulatory interaction with the
population of interest, and may have population lists. For example, state Departments of Transportation or Motor Vehicles may have
lists of auto salvage yards.
There are some data collection companies that primarily serve the financial sector. The most prominent among these companies are
Dun and Bradstreet (D&B) and Standard and Poors' (S&P) Compustat. The data from both of these covers a large number of private
and public companies. The companies do not, however, tend to cover smaller companies. Nevertheless, these sources may be useful.
Contacting either company and discussing the issue with a sales representative will assist you in determining if they can be of
assistance. (Dun and Bradstreet: 800-624-5669; Compustat: 800-523-4534).
All states require some form of licensing for businesses to operate. You should consider contacting these departments to determine
the level of information they can provide. Some states are more willing than others to share information.
-------
Appendix G
Page G-5
Table 1 (continued)
Source
Comments
Direct Marketing Lists
Several companies offer lists of businesses that can be used in direct marketing campaigns. These companies draw their lists from
several sources (e.g., Yellow Pages, directories). Some of them offer customized services. These companies typically charge a fee
for lists; the fee should be weighed against the benefits of quickly obtaining reliable lists. Before using such a company, however,
you should review the sources that the company uses and the likelihood that any list you purchase will match your target group. A
good resource for finding these companies is the DM News web site (http://www.dmnews.com). The site contains a "Yellow Pages"
of direct marketing companies that lists companies by the services they provide. We suggest companies in the "List
Broker/Manager/Compiler" or "Mailing List Broker" categories as a starting point.
Drive-bys for limited
population sizes and
areas
One manual means of collecting data is through a "drive-by" approach. If you are interested in a limited geographic area (e.g., a
specific city, a specific body of water) then collecting the population information by physically enumerating them may be possible.
That is, you go out into the field and record each population member that you come across. This process, however, can become
complicated as the size or the geographic area increases in size or complexity. Nevertheless, if you can efficiently cover the
geographic area and can accurately identify population members, then this approach may be an effective means of developing a list.
Equipment
manufacturers and
suppliers
Most businesses require some form of equipment and supplies to operate. For example, many auto repair shops purchase parts from
auto parts dealers/retailers. Some equipment manufacturers and suppliers may be willing to provide detailed customer lists. These
lists may provide a basis for a population list. The key here is to find manufacturers or suppliers that provide specialized equipment
(i.e., used only by the population you are interested in) and that provide the equipment to a large number of the population. There
are some considerations here, however. Specifically, some members of the population who purchase from these sources may have
asked not to have their names and information passed on to third parties. Another consideration is that not all members of the
population may purchase from the same manufacturer or supplier. Nevertheless, if a manufacturer or supplier is willing to share a
customer list, they may also be willing to discuss the completeness and accuracy of that list.
-------
Page G-6
Appendix G
Table 2. Some Sources That Are Not As Valuable For Building Population Lists
Source
Marketing-Oriented Lists
Better Business Bureau
Chambers of Commerce
U.S. Census Bureau
Trade Show Participants
Comments
Several sources offer lists of companies providing a specific service. For example, you may find a list of "dry cleaner
supply companies" that has been distributed to dry cleaners. You should determine how that list was compiled. There are
cases where lists are compiled for marketing purposes. Inclusion on such lists may require payment. This would mean that
these lists are biased towards those that are willing to pay to be on the list.
The Better Business Bureau (BBB) is a valuable resource for a number of things, but not as a source for population lists.
BBB data do not include sufficient detail and rely on a number of potentially biasing criteria.
As with the BBB, Chambers of Commerce offer valuable resources to the business community. Chambers of Commerce
are not, however, good sources of population lists since membership in these organizations generally requires payment of
dues as well as other requirements.
The U.S. Census Bureau is an excellent source for aggregated data, but is not a source for company-specific data. The
Census will not release any information that can identify a specific company. Census Bureau data is useful, however, in
estimating the population size as we will discuss below.
Companies that have participated in trade shows may seem like a good starting point, but refining the list may be more
trouble than it is worth. Trade shows offer a meeting place for a specific service or product category. Even if you are able
to find one that caters specifically to your target group, be aware that not all participants will be in the target group. Many
participants may be offering services or products that are used by your target group. Trade show participant lists may be
useful to have, but may not provide useful input into a population list.
-------
Appendix G Page G-7
III. Estimating the Population Size
When dealing with unknown populations, developing an estimate of the population size is an important
first step. The problem, however, is that there is no standard method for estimating the population size in
cases of unknown populations. This section presents ideas on how to estimate population sizes using
available information. We offer three approaches to estimating population size and a method of
combining the three approaches to develop a "reasonable range" for your population size.
Approach 1: Use the Number in the List You Compiled—Your first consideration should be to
assess the completeness and accuracy of your final list. For this assessment, consider how each
individual source developed its own list, determine which segments of your population may be missing
from your final list, and identify if out-of-scope entities are in your list. If you can say with some
certainty that your final list is relatively complete and accurate, then your best estimate of your
population size is the number of entities in your list. This is rarely the case, however, with unknown
populations.
Approach 2: Use Source Estimates of Coverage to Generate a Population Number—A second
approach to estimating the population size is to discuss the accuracy of a source with the organization
that developed the source. This is certainly more productive with some sources (e.g., trade associations)
than with others (yellow pages). Organizations such as trade associations or marketing data services
may have already developed estimates of their coverage. For example, a trade association may be able
to tell you that they cover 75 percent of all businesses in the sector. From there you can make an
estimate of population size.67 You should obtain an estimate of coverage from each source that can
provide a reasonable estimate of coverage (e.g., the phone company may not be able to provide such an
estimate for the yellow pages). If you can do this for more than one source, then look at the range of
those estimates. If that range seems reasonable, then the range can work as a feasible estimate of the
population size.
Approach 3: Census Bureau Data—A third means of the estimating the population size is to use data
collected by the U.S. Census Bureau. The U.S. Census Bureau collects information on businesses
operating the in United States. The Census data that may be most useful for regional and state-level
performance measurement projects would be the County Business Patterns data,68 which presents the
numbers of business establishments for various sectors at the state, county, and metropolitan statistical
area (MSA) levels. For example, using this data for 1999, there are 3,828 "automotive repair and
maintenance shops" in Massachusetts, of which 89 percent employ less than 10 employees. Sectors are
defined using the North American Industrial Classification System (NAICS) which provide a great
amount of detail. For example, "automotive repair and maintenance shops" are further divided into
mechanical and electrical repair and maintenance and auto body repair. The mechanical and electrical
grouping is then further divided into other groups. The census web site also offers detailed definitions
of the NAICS, allowing you to determine which ones are relevant to your project.
Approach 4: Combination of above three —Finally, you can combine all three of the above
approaches and formulate a "reasonable range. " A range is possibly the best you can do for cases of
unknown populations. A reasonable range should include the actual population size, while a single
estimate risks being far away from the actual value since we have little information on the actual
67 In this case, the estimate is to take the number in the trade association list and divide by 0.75.
68 The Census County Business Patterns data can be accessed online at
http://www.census.gov/epcd/cbp/view/cbpview.html. It is available in an online database, HTML, and PDF formats.
-------
Page G-8 Appendix G
population size. Thus, we suggest using a range as your size estimate unless you are relatively certain
you are close with a single estimate.
To formulate the reasonable range, we suggest you write down all of the estimates that you have made
O 5 OO J J
from the above-mentioned approaches. Then, rate each on a scale of one to five on how confident you are
in that estimate, with five being the most confident. You can then use the following rules to develop a
"reasonable range":
If you have more than one "five"—Use the smallest and largest estimates with a five rating as your
range.
If you have one "five"—Use your most confident value below and above the estimate rated as "five"
as your range. Describe it as a range with a "most likely value" equal to the five-rated value. If the five-
rated value is not within the range, use it as either the upper or lower bound of the range and describe it
as "a range of x to y, but most likely closer to y" (assuming y was the five-rated value).
If you have no "fives"—Use your most confident lowest and highest values as your range.
You should try not to use any "ones" or "twos" in developing your range. For example, in the second rule
above, you may be better effusing the five-rated value as an upper or lower bound rather than using a
"one" or "two" as the upper or lower bound. If applying this general rule, however, results in only one
remaining value, then you are better effusing some "ones" and "twos" than having only a single estimate.
This method is not based on theoretical statistical concepts, but is intended for someone with limited
statistical background. Statisticians have developed means of estimating population sizes for unknown
populations based on theoretical statistics. However, unless you have access to a qualified statistician,
successful implementation of these ideas may be impossible. Also, the three approaches used for this
method should not be considered the only means of estimating population size. Any other approach that
provides a reasonable estimate is worthwhile exploring.
-------
Appendix G Page G-9
IV. Sampling Methods
This section discusses three informal sampling methods that can be used to select members of the
population for inclusion in your study: judgement sampling, quasi-random sampling, and purposive
sampling. We call these informal methods because they do not satisfy the requirements of random
sampling. This does not, however, imply that these methods provide invalid or misleading results.
Informal sampling methods can provide useful and insightful information at less cost than statistically
valid methods. The ability to extrapolate the results from informal methods to a larger population,
however, is limited since you cannot ensure the representativeness of your sample. For each method, we
assume that you have been able to compile a fairly reliable list and that you have made some estimate of
the population size. We provide some examples of how to conduct these type of sampling exercises in
Table 3. Following our discussion of the three sampling methods, we provide a short discussion of
choosing a sample size.
Judgement Sampling
Judgement sampling involves choosing a sample based on your ideas of who should be in the
sample. This method of sampling is the farthest from statistically valid random sampling since it involves
a subjective choice for selecting the sample. The advantage of this method, however, is that when
choosing the participants, you can ensure that your sample contains respondents who can provide
interesting or detailed information. For example, you may select a facility that you know has a detailed
environmental management system and has addressed issues in which you are interested. You may want
to choose facilities that fit a certain profile (e.g., large facilities that perform a specific operation). The
key here is that you set the criteria for inclusion based on specific reasons.
The problem with judgement sampling is that the sample you obtain will not represent the entire
population. Thus, taking information from your judgement sample and extrapolating this to your
population is not valid or reliable. On the other hand, by specifying who you talk to, you can maximize
the amount of information you obtain. This type of sampling method can assist you in defining issues,
identifying problems or solutions, and understanding processes.
Quasi-Random Sampling
A quasi-random sample involves choosing a subjective criterion but using a probability-based
mechanism for choosing participants. The subjective criterion can be any number of things, but it
usually has to do with convenience. For example, a quasi-random sample may be a survey taken of
seminar participants that asks questions beyond the topics of the seminar. Another example of a quasi-
random sample would be to query every 10th member of the population.
Quasi-random samples are not representative and may provide results that are biased in an unknown
manner. Quasi-random samples, however, are a less-costly means of selecting a sample than a judgment
sample, and are also useful when you have little information on who to choose in a judgement sample.
-------
Page G-10
Appendix G
Table 3. Examples of the Three Informal Sampling Methods
Method
Examples
Best Applications
Judgement Sampling
• Choose "best performing" facilities.
• Choose facilities that have compliance problems.
• Choose facilities that have been active in other Agency
programs.
• Choose facilities that perform specific operations.
• Choose facilities that have been helpful in the past.
Judgement sampling is best suited for situations where you
need specific or specialized information. That is, you are less
concerned with obtaining a representative group of
participants, but more concerned with selecting a group that
can tell you specific things (e.g., how a compliance
assistance program improved environmental performance).
Quasi-Random Sampling
• Choose every w-th (e.g., tenth, fifth) on the list.
• Randomly sort your list using a random number generator
(e.g., from a spreadsheet) and choose participants from the
start of the sorted list until you reach your target number of
participants.
• Choose seminar participants as your sample.
• Choose facilities that are convenient for on-site interviews.
• For a mailed survey, choose some of the facilities on your
list.
Quasi-random sampling is best suited for situations where
you want to obtain a somewhat representative sample, but do
not have the resources to develop a statistically valid
approach. A quasi-random sample sacrifices some
randomness (and hence representativeness) to reduce the
cost of the collection. Nevertheless, some effort is made to
obtain a sample that is representative, given the resource
constraint.
Purposive Sampling
• Divide the population between some relevant groups (e.g.,
between small and large) and choose a sample that mimics
that division. For example, if 20 percent of the entire
population is small, then be sure 20 percent of your sample
is small).
• Choose some facilities with poor compliance records and
some with stellar compliance records. Divide the sample
evenly between the two groups, if possible.
Purposive sampling is best suited for cases where one or
more factors (e.g., size, compliance status, use of a specific
compliance assistance tool) is important. In most cases
where purposive sampling is used, you are making sure that
you have some information on each "side" of the factor (e.g.,
both large and small) because you are particularly interested
in exploring differences between the two "sides."
-------
Appendix G PageG-11
Purposive Samples
A purposive sample uses objective selection criteria, but a non-probability mechanism to choose
participants. In a purposive sample, you are looking to replicate some aspect of the population, which can
be very complicated (e.g., trying to replicate statistical parameters). One form of purposive sampling
involves replicating population characteristics. For example, you may know that 80 percent of all
facilities are "small" facilities. If you had a sample size of 20, you would choose four "large" facilities
and 16 "small" ones.69
Purposive samples are more difficult to select and require more information on the population.
However, purposive samples are the least biased and are closer to statistically valid than the other two
methods. They are not, however, representative since they do not satisfy the requirements of statistical
validity.
Choosing A Sample Size
Given that these methods are not statistically valid, the number of participants that you choose can be
based on resource constraints. Naturally, as the number of participants rises, the more reliable your
results. To determine a sample size, we suggest that you determine the sampling method you will use and
how you will collect your data (e.g., mailed survey, interviews). Determine the resources that you will
require per participant, and then determine the total number of participants that you can afford based on
resource constraints. Try for that maximum number, but be aware that the number you choose has no
statistical grounding. That is, if you come up short of the number you are not violating any statistical
properties of your sample.
V. Summary and Conclusion
This guide has provided information on developing population lists and choosing samples from
unknown populations. The fact that a population is unknown or hard to enumerate should not deter you
from collecting information. Unknown populations require you to take the first step of generating a
reliable population list, rather than relying on a readily available one. We expect that this guidance will
assist you in developing such lists. In cases of unknown populations, developing a population list can be a
valuable project outcome in itself even if that list is known to be only partial. What is most important in
dealing with unknown populations is to analyze and evaluate your data sources at each step. Constant
analysis and evaluation will help you understand the limitations of your final product.
Your final list will probably not contain all of the population members and will probably contain
entities that are not in your population. We expect that this guidance will help you minimize as much of
this over- and under-coverage as possible. Nevertheless, even well-funded statistical sampling projects
often rely on less-then-perfect population lists. As we have stressed in this guide, however, the key is to
understand the sources of the imperfections. Understanding the imperfections of your list will go a long
way to help you understand the imperfections and limitations of data you collect using your list.
69 There is no real restriction on how to choose the facilities at this point. You can, in fact, use one of the other
informal methods such as a judgement sample. You can also randomly select the facilities from among the "small"
and "large" ones.
------- |