SEPA
United States
Environmental Protection
Agency
EPA 902-R-9 9-002
August 1999
Customer Service
in Permitting
A Toolkit for Regions, States, Tribes,
and Local Permitting Authorities
-------
DISCLAIMER
This Toolkit is intended only as guidance. It is not binding on EPA or any other person
and is not intended to create any rights enforceable by anyone. While Regions, States, Tribes, and
local permitting authorities are encouraged to consider and use the material in this Toolkit, they
are free to act at variance with it.
-------
Acknowledgements
This document was conceived and prepared by the national Customer Service in Permitting
Workgroup established by the U. S. Environmental Protection Agency's Region 2 Administrator
Jeanne Fox. She served as co-lead of the permitting core process under the EPA Customer Service
Steering Committee with Timothy Fields, Acting Assistant Administrator for Solid Waste and
Emergency Response.
Membership and level of participation in the workgroup included, at various times,
representatives from EPA Headquarters, EPA Regions 1, 2, 4, 5, 6, and 8, and more than 20 states.
We want to thank all contributors to this effort, but, in particular, acknowledge the following core
members of the workgroup without whose dedication and expertise this document could not have
been produced:
Stanley Siegel, EPA Region 2, Division of Planning & Protection (Workgroup Chair)
Andrew Bellina, EPA Region 2, Division of Planning & Protection
Patricia Bonner, EPA HQ, Office of Policy
Steve Burkett, EPA Region 8
Patricia (Tricia) Buzzell, EPA HQ, Office of Solid Waste & Emergency Response
Lance Miller, EPA and State of New Jersey
Vernon Myers, EPA HQ, Office of Solid Waste & Emergency Response
Joel Salter, EPA HQ , Office of Water
Leo Stander, EPA HQ, Office of Air Quality Planning & Standards
Peg Anthony, Macro International, Inc.
-------
Introduction
About this Toolkit: Customer
Service in Permitting
Permits are a cornerstone of environmental protection because they spell out how regulated
communities must comply with environmental laws. EPA and its regulatory partners (states, tribes,
and local governments) issue permits to control facilities' emissions into the air and water, and to
ensure safe management of hazardous waste.
Many people have their first encounter with an environmental agency through the permitting process.
That makes customer service a particularly important part of permitting. Our customers include the
general public, individuals who may be affected by the permit decision, state, tribal, and local
governments authorized by EPA to issue permits, and permittees that need permits to operate.
Early in 1999, EPA announced its "Next Generation in Permitting," an action plan to move permitting
towards measuring performance, while providing regulated parties with more flexibility in how they
meet standards. This action plan is part of EPA's broad-based agenda for reinventing environmental
protection, and it builds on the work of EPA pilot programs that have tested innovative approaches
to permitting, including Project XL. The action plan and other permitting reform efforts show that
EPA is committed to:
• Strengthening the role of the public in important decisions
• Focusing on results instead of procedures
Reducing unnecessary burden
As part of our plan to reinvent environmental permitting, EPA committed to preparing a user-friendly
toolkit of customer service processes and techniques for use by EPA and its partners. This toolkit,
which is also part of an Agency-wide Customer Service Strategy, is the result.
Through a workgroup that included EPA Headquarters, Regions, and states representatives, EPA has
endorsed four broad customer service standards for environmental permitting. These standards are
discussed in Chapter 3 of this toolkit:
We will prepare permits that are clear, fair, appropriate, enforceable, and effective.
• Our staff will be knowledgeable, responsive, cooperative, and available.
• We will work with representatives of permitting authorities to continually improve permitting
processes and services.
• We will make our permit decisions within the time frames that are established for the type of
permit being requested.
i- 1
-------
These permitting standards have been adopted by EPA. Other permitting authorities are encouraged
to adopt these or similar permitting standards designed for their organization. This toolkit is intended
to help staff at EPA, as well as other permitting authorities meet these customer service standards.
Specifically, it offers tools for obtaining customer feedback and using it to improve permitting
processes to meet customer needs. After this introduction, the Toolkit is organized to provide
information on:
• A Customer Service Orientation - what is EPA's customer service program and what this
toolkit is about
• Partners in Permitting Service Delivery — who the permit process participants are and how
they interrelate as customers-suppliers
• Permitting Standards — setting standards by which we can gauge good customer service
• Feedback and Measurement — planning for and gathering of feedback
• Responding to Feedback — analyzing and acting upon feedback data
• Maintaining Good Customer Service — creating the cultural change necessary to keep
customers satisfied, permit after permit
• Customer Service in Action — examples of what other EPA regional offices and states have
done
As with any toolkit, readers of Customer Service in Permitting will need to decide for themselves
which tools are most appropriate for their particular circumstances.
-------
Table of Contents
Page No.
Introduction i - 1
Chapter 1: A Customer Service Orientation 1- 1
What Is EPA's Customer Service Program? 1- 1
So How Will Things Be Different Now? 1 - 1
Who Should Use this Toolkit - and Why? 1-2
What Are Our Services? 1-2
Who Are Our Permitting Customers? 1-3
How Is Quality Related to Services Provided? 1-4
Chapter 2: Partners in Permitting Service Delivery 2- 1
What Is the Permit Process? 2 - 1
Who Are the Permit Process Participants? 2- 1
Who Are the Permit Service Providers? 2- 1
What Is the Relationship Between the Customers and Suppliers? 2-1
What Are the Services Being Delivered and How Are They Evaluated? 2-1
What Do I Do With The Following Tables? 2-2
Chapter 3: Permitting Standards 3- 1
What Are Customer Service Standards? 3- 1
What Are EPA's Permitting Customer Service Standards? 3- 1
How Might a Permit Service Provider Fulfill These Standards? 3- 1
Chapter 4: Feedback and Measurement 4 - 1
Why Do I Need Customer Feedback? 4- 1
Why Should I Set Customer Service Standards and Goals? 4- 1
How Do I Design a Feedback & Measurement System? 4-2
Plan the Customer Feedback Procedure 4-3
How Ready Is My Organization for Customer Feedback? 4-3
What Kinds of Customer Feedback Are Already Occurring? 4-4
What Are the Core Questions I Should Ask of My Customers? 4-4
How Often Should We Ask Customers for Feedback? 4-5
How Long Should Feedback Activity Take? 4-6
Why Should I Establish Quality Control Procedures? 4-7
Construct Data Collection Procedures 4-9
What Is the Best Approach for Assessing Customer Satisfaction? 4-9
Continuous Assessment 4-9
Decide on Data Collection Method 4-9
The Sample 4-13
-------
Determine the Sample Size 4-13
Develop the Questions 4-14
Construct the Questionnaire 4-17
Mail Surveys 4-17
Focus Groups 4-18
Telephone Surveys 4-19
Pretest 4-20
Contingency for Non-response 4-21
OMB Clearance (EPA Only) 4-22
Model Survey Instruments 4-23
Additional Resources 4-23
Conduct Data Collection 4-24
Focus Groups 4-24
Mail Surveys 4-25
Telephone Surveys 4-26
Electronic Feedback 4-27
Chapter 5: Responding to Feedback 5 - 1
Analyze the Data 5-2
Data Clean-up 5-2
Types of Data and Analyses 5-2
Analysis: An Example 5-3
Driver Analysis 5-8
Presenting the Data 5-8
Making Recommendations Based on the Data 5-9
Presenting Recommendations - Using Graphics 5-9
Act on the Results 5-10
Is This the Beginning or the End of the Process? 5-10
How Do You Decide What to Do with the Feedback You Receive? 5-10
How Good Is Good Enough? 5-10
How Do We Know What to Work on First? 5-11
Chapter 6: Maintaining Good Customer Service 6 - 1
Chapter 7: Customer Service in Action 7-1
Introduction 7-1
Permit Assistance/Information Center 7-1
Stakeholders' Early Participation in the Permit Process 7-2
Plain Language 7-2
Permit Processing/Issuance Timeframes 7-3
Customer Satisfaction Surveys 7-3
Incentive/Award/Reward Programs 7-4
Appendix A: Internal Control Procedures
Appendix B: Sampling
-------
Appendix C: OMB Clearance
Appendix D: Model Surveys
i-5
-------
Chapter 1
A Customer Service
Orientation
What is this toolkit about? Who is it for and why should it be used? These
questions and other general questions about service to customers are
addressed in this first chapter of the Customer Service in Permitting Toolkit.
What is EPA's Customer Service Program?
On September 11, 1993, President Clinton issued Executive Order 12862, Setting Customer Service
Standards," to all customers.
The order led EPA to adopt the following vision for the Agency's customer service efforts:
As we achieve our mission of protecting public health and the environment, EPA people are
becoming customer-focused, our products and services customer-driven, and our customers
satisfied.
EPA's Customer Service Strategy includes seven elements:
• Setting customer service standards
• Publicizing the standards
• Establishing measures and tracking systems and providing measurement assistance
• Building staff capacity by making training and information available
• Involving staff in the development of customer service programs
• Benchmarking EPA against world class service
• Providing managers assistance and actionable information
How Will Things Be Different Now?
As a customer-focused agency, we hold ourselves accountable for providing service that rivals the
best in the private sector. We have a set of standards against which we measure ourselves, the Six
Principles of Customer Service:
1) Be helpful! Listen to your customers.
2) Respond to all phone calls by the end of the next business day.
3) Respond to all correspondence within 10 business days.
4) Make clear, timely, accurate information accessible.
5) Work collaboratively with partners to improve product and services.
6) Involve customers and use their ideas and input.
1 - 1
-------
To implement these principles, EPA identified eight core processes that are most important to our
customers:
• Permitting
• Enforcement and Compliance Assistance
• State, Tribal and Local Grants
• Partnership Programs
• Public Access
• Pesticides Registration
• Research Grants
• Rulemaking
The workgroup prepared this Permitting Toolkit to help you improve customer service within the
permitting core process.
Who Should Use This Toolkit - and Why?
This toolkit is not just for EPA. Many non federal environmental agencies helped us make it a
resource for any permitting authority — states, tribal, local, or federal. It can help all of us to provide
better environmental and public health protection.
Like any toolkit, it is not a mandate, but a set of tools that all permitting programs can use. The
toolkit describes how you can gather reliable feedback from your customers, and how some agencies
have redesigned their permitting programs to meet customer needs. Asking customers about their
needs, expectations, and experiences enables us to measure whether they are satisfied with our
services. And customer satisfaction can save resources in the long run by avoiding unnecessary and
time-consuming confrontation.
The bottom line? Finding out customer opinions about what we do and how we do it will help us
improve our products and services. Customers will notice and value these improvements, learning
that government agencies can be more responsive to customer needs.
What Are Our Services?
Permits describes what facilities must do to meet environmental health and environmental safely
standards. In many cases, EPA authorizes or delegates state, tribal, and local governments to issue
permits. (We will refer to both these in this Toolkit as "delegating.") In jurisdictions where
delegation has not occurred, EPA issues permits directly. Permits control facility emissions into the
air (e.g., permits to modify or construct, and operating permits), protect surface and groundwater
(e.g., national pollution discharge elimination, storm water runoff, underground waste injection) and
ensure the safe management of hazardous waste (e.g., treatment, storage and disposal).
1 -2
-------
Who are Our Permitting Customers?
EPA's Hearing the Voice of the Customer - Customer Feedback and Customer Satisfaction
Measurement Guidelines define a customer as someone who directly relies on a provider for a
product or service. Customers are defined on the basis of the service or product they receive. In
permitting, a customer may:
• have a direct relationship with the permitting authority
• receive one or more services or products from the permitting authority
• be directly affected by the actions of the permitting authority
• be an employee of the permitting authority, acting as an internal customer
While it is possible to identify and label many customer groups that are interested in the permitting
process, this toolkit focuses on two major groups — "interested and impacted parties" and "permit
applicants." Interested and impacted parties are those individuals, interest groups, communities,
states, or tribes that raise a concern or have comments regarding the permit action. Permit applicants
are seeking approval from EPA or a delegated authority to conduct a regulated activity.
This toolkit describes the relationships among the interested and impacted parties, the permit
applicants, and the permitting authorities. We also briefly discuss internal relationships among the
permitting authorities (EPA headquarters, regional offices and delegated authorities).
Table 1.1 outlines the relationship between the customers and their service providers in the permitting
process.
Table 1.1
Customers and Their Service Provider
Customer
EPA Regional Office
Delegated State, Tribal or Local
Government
Permit Applicant
Interested and Impacted Parties
Service Provider
EPA Headquarters
EPA Regional Office
EPA Regional Office or Delegated State,
Tribal or Local Government
EPA Regional Office or Delegated State,
Tribal or Local Government
Because these customers often have differing and conflicting needs, delivering customer service in
permitting becomes a complicated matter. However, as stated in the "Blair House Papers" (January
1997), an agency desiring to be a customer-driven organization must continuously ask its customers
1 -3
-------
what they want. Experience shows we can identify the needs of the permitting customers and work
to satisfy those needs while still carrying out our mission to protect human health and to safeguard
the natural environment.
How is Quality Related to Services Provided?
Figure 1.1 shows that customer satisfaction depends both on the services we deliver and the way we
deliver them. Obviously, we should deliver services in a "right way" rather than in a "wrong way."
But, perhaps you haven't considered that the services you provide may not be the ones your
customers want. Imagine the reservoir of potential services that go untapped because we are out of
touch with our customers, spending unnecessary time doing the wrong things, or doing the right
things in a wrong way. A customer-focused organization should know whether its services meet
customer needs, and take advantage of opportunities to invest in newer and better services.
Figure 1.1 Services and
Delivery Model
How it's done
Right
Wrong
Services
What's done
Wrong Right
Right Things
Done Right
(Satisfied Customers)
Wrong Things
Done Right
(Bewildered
Customers)
Right Things
Done Wrong
(Irate Customers)
Wrong Things
Done Wrong
(Disenchanted
Customers)
Missed opportunities to do
more important things
because we are out of sync
with our customers
1 -4
-------
""-. :t"" Id; ir I! |i |; iUI.M l\
-::=- - «,>. -„--,-,, %::,«^.«. ,-„ ». -^,.,^5, ^-JB ;|
Chapter 2
Partners in Permitting
Service Delivery
To provide outstanding customer service, you must understand the dynamics of
customers, suppliers and services. This chapter identifies the who of the permitting
process — those providing services and those receiving them — and the what of the
permitting process — the services being delivered.
What is the Permit Process?
The permit process encompasses all the steps in making permitting decisions, including how you write
guidance or regulations, obtain and review information, oversee state and local partners, seek public
input and make decisions. The ultimate product is official approval or denial for a given permit or
permitting program.
Who are the Permit Process Participants?
• EPA Headquarters
• EPA's 10 Regional offices
State, tribal, or local government
• Permit applicant
• Interested and impacted parties
Who are the Permit Service Providers?
The permit-service providers are EPA Headquarters, EPA Regions, and a delegated governmental
permitting authority, where applicable. Permit service providers either make the permit decision, or
provide services to the decision-making organization.
What is the Relationship Between the Customers and Suppliers?
The relationship between the permit service providers and their customers is shown in Figure 2.1.
The solid lines of this figure represent services provided to the customers. The dotted lines indicate
2- 1
-------
opportunities for obtaining customer feedback on those services.
What are the Services being Delivered and How are they Evaluated?
Tables 2.1 through 2.4 show the types of customer feedback you should seek for each service you
provide. Don't forget to ask your customers about additional services they need that you're not
delivering.
There are two special relationships shown in Figure 2.1. First is the direct feedback from delegated
states to EPA Headquarters through the Environmental Council of the States (ECOS) and other state
organizations. This relationship, which bypasses the Regions, often deals with higher level, non-
permit-specific issues. Second is the community-based relationship between the permit applicant and
the interested and impacted parties. We envision a relationship characterized by the Community
Based Environmental Protection (CBEP) program, where all interested parties develop a joint
environmental vision and processes to achieve it.
What Do I Do With The Following Tables?
Look at the table that represents your customer-supplier situation and the services being provided.
Then look at the delivery feedback needed to assess whether you are asking for the appropriate
customer feedback, i.e., whether you are customer-focused. The feedback surveys of Chapter 4
should help you get this important input from your customers.
You can also "turn" these tables. When you are a customer, you should have certain expectations
for the services delivered to you. If your expectations are not being met, perhaps you can start a
dialogue with your service provider.
2-2
-------
Figure 2.1
Relationships,
Services and
Feedback
•Surveys
•Interviews
•FTE and $
•Regulations
•Policy
•Guidance
•Initiatives
•Interpretations
•Program Reviews/
Evaluations
( Regional ^
( ° ) <
V Office J %
•ECOS
•Other State
Organizations
•Resources
•Policy
•Guidance
•Initiatives
•Interpretations
•Program Reviews/Evaluations
•Delegations
'Surveys
Interviews
s
/ Delegated
\Authority
Provided by either the Regional
Office or Delegated Authority
•Regulations
•Policies
•Guidance
•Forms/Forums
•Interpretations
•Timely Permit Process
•Access to Information
f Permit
V Applicant
"""• /"^ ^^\
^Interested X
( & Impacted )
CBEP
Service
Feedback
2-3
-------
Table 2.1
Delivery Feedback Needed for Services Delivered
Service Provider: EPA Headquarters
Service Customer: EPA Regional Offices
Services Provided
Resources
Regulations
Policy
Guidance
Initiatives
Interpretations
Program Reviews and
Evaluations
Delivery Feedback Needed
Was the process used to allocate resources to the Regions fair and reasonable? Do
Regions want more input to the allocation process?
Do Regions have the regulatory tools to carry out their permit programs? Seek input
on regulatory reform needed.
Do Regions have the policies needed to carry out their permit programs? Are existing
policies clear and useful?
Do Regions have the necessary guidance to carry out their permit programs? Is
existing guidance clear and useful?
What initiatives do the Regions need to carry out the permit programs? Which
initiatives are counterproductive, ignored or unwanted? Seek input on initiatives'
usefulness.
Do Regions receive timely and accurate interpretations? Assess quality of
interpretations to determine whether you are meeting regional needs and expectations.
Do Regions receive timely and accurate program reviews and evaluations? Are they
useful? Assess whether you are meeting the Regions' needs and expectations.
2-4
-------
Table 2.2
Delivery Feedback Needed for Services Delivered
Service Provider: EPA Regional Offices
Service Customer: Delegated Authorities (State/Tribal/Local Governments)
/Regional \
Office
Services Provided
Resources
Regulations
Policy
Guidance
Initiatives
Interpretations
Delegation or
Authorization of
Programs
Program Reviews and
Evaluations
(Oversight)
Delivery Feedback Needed
Was the process used by EPA to allocate resources fair and reasonable? Do delegated
authorities want more input to the allocation process? Is NEPPS providing flexibility
for the delegated authorities to allocate resources where needed?
Are the federal regulatory tools appropriate and adequate to carry out the programs?
Seek input on regulatory reform needed.
Do delegated authorities have the national or regional policies they need to carry out
their permit programs? Are existing policies clear and useful?
Do delegated authorities have the guidance they need to carry out their permit
programs? Is existing guidance clear and useful?
What initiatives do the delegated authorities need to carry out the permit programs?
Which initiatives are counterproductive, ignored or unwanted?
Do delegated authorities receive timely and accurate interpretations? Assess quality of
interpretations to determine whether you are meeting delegated authorities' needs and
expectations.
Are delegations/authorization application instructions clear? Is the processing of
delegations/authorizations timely? Seek input on experiences with
delegations/authorizations.
Do delegated authorities receive timely and accurate program reviews and
evaluations? Are they useful? Do they meet the needs and expectations of delegated
authorities?
2-5
-------
Table 2.3
Delivery Feedback Needed for Services Delivered
Service Provider: Permit Authority (EPA Regional Office or Delegated Authority)
Service Customer: Permit Applicant
Services Provided
Regulations
Policy
Guidance
Forms
Interpretations
Timely Permit Process
Delivery Feedback Needed
Are regulations understandable and reasonable? Do they meet the permit applicant's
needs? Seek input on regulatory reform needed and forward it to EPA Headquarters,
as appropriate. Such feedback may lead to industry-wide or individual facility
reinvention activities, such as the Common Sense Initiative (CSI), ECOS Innovations
Agreement, or the XL program.
Are program policies understandable, reasonable and supportive of permit applicants'
needs?
Is permit guidance understandable, reasonable and supportive of permit applicants'
needs?
Are permit application forms and instructions understandable, reasonable and
supportive of permit applicants' needs?
Do permit applicants receive timely and accurate interpretations that meet their needs?
Are permit decisions timely? Assess permit applicants' needs for timeliness.
2-6
-------
Table 2.4
Delivery Feedback Needed for Services Delivered
Service Provider: Permit Authority (EPA Regional Office or Delegated Authority)
Service Customer: Interested and Impacted Parties
Services Provided
Regulations
Policy
Guidance
Opportunities for
Involvement
Forums
Access to Information
Interpretations
Delivery Feedback Needed
Are regulations understandable and reasonable? Do they meet the needs of interested
and impacted parties? Seek input on regulatory reform needed and forward it to EPA
Headquarters.
Are program policies understandable and reasonable? Do they meet the needs of
interested and impacted parties?
Is permit guidance understandable and reasonable? Does it meet the needs of
interested and impacted parties?
Do these parties have ample opportunity to be involved in the permitting process?
What additional involvement do they need?
Are forums for public input and interaction reasonable and supportive of these
customers' needs? What additional forums do they need?
Do these customers have access to the information they need to review and comment
on the permit applications?
Do these customers receive timely and accurate interpretations? Are the
interpretations meeting their needs and expectations?
2-7
-------
Chapter 3
Permitting Standards
has established customer service standards for its own permitting work
that can serve as a guide to other permitting organizations. This chapter deals
with permitting standards — why it is important to have them and measure
them, and how permit service providers can fulfill them.
What Are Customer Service Standards?
EPA's customer service standards establish a yardstick by which the Agency will measure itself. EPA
has grouped these standards into 9 groups: Universal; Public Access; Partnership; State/Tribal/Local
Program Grants; Pesticide Registration; Enforcement Inspections and Compliance Assistance;
Rulemaking; Research Grants; and Permitting. EPA created the Customer Service in Permitting
workgroup to develop the tools needed to meet the customer service standards for permitting,
including ways we can obtain feedback from our permitting customers. Much of our work is
summarized in this toolkit.
What are EPA's Permitting Customer Service Standards?
We took draft standards developed by the Agency and worked them into this final set of standards:
We will prepare permits that are clear, fair, appropriate, enforceable, and effective.
Our staff will be knowledgeable, responsive, cooperative, and available.
We will work with representatives of permitting authorities to continually improve
permitting processes and services.
We will make our permit decisions within the time frames that are established for the
type of permit being requested.
We believe these standards, should lead to world-class customer service. However, we encourage
other permit-issuing organizations to modify these standards as needed to fit their particular situation
and customer expectations.
How Might a Permit Service Provider Fulfill These Standards?
Each of the permit service providers listed in Chapter 2 has a role in meeting the four permitting
customer service standards. Table 3.1 illustrates some activities that might fulfill these standards.
It is important to ask customers what they want before deciding how your organization will meet the
standards. That will be the focus of Chapter 4, Feedback & Measurement.
3 - 1
-------
Table 3.1
Fulfillment of Permitting Customer Service Standards by Service Providers
We will prepare permits that are clear, fair, appropriate, enforceable, and
effective
Headquarters
• promulgate permit regulations that are understandable, written in plain English, and
workable
• write guidance materials that clarify the intent behind the permitting requirements, and the
rationale leading to final regulatory decisions
• make interpretations on a case-by-case basis whenever questions arise
• stress consistency in application to take the guess work out of implementation
• seek feedback from customers on whether Headquarters' services are clear, fair,
appropriate and effective
Regions
• write guidance in plain English
• tailor training programs to meet the needs of delegated permitting authorities
• make permit oversight reviews productive and to the point, taking into account
circumstances specific to the permit; do not nit-pick
• seek feedback from customers on whether the Region's services are clear, fair, appropriate
and effective services
State/Tribal/Local (and Region, if Region is directly implementing a permit program)
• write permits that reflect the unique nature of the permit applicant's situation
• seek input from the permit applicant and other interested and impacted parties
• make permit decisions after carefully weighing all of the input from the permit applicant
and the interested and impacted parties
• seek feedback from customers on whether the permitter's services are clear, fair,
appropriate and effective
Our staff will be knowledgeable, responsive, cooperative, and available.
Headquarters
• prepare training programs covering all appropriate rules and regulations for regions, states
and regulated community
• respond to inquiries from permit writers in a timely fashion and in the appropriate level of
detail
respo
level of detail
3 -2
ea
• respond to correspondence, hotlines inquiries, etc., in a timely fashion at the appropriate
level of detail
-------
• maintain records on policy memoranda that are used to interpret regulations, thus ensuring
that permit writers have consistent information
• seek feedback on whether Headquarters' staff is knowledgeable, responsive, cooperative,
and available
• to the extent practicable, maintain key policy memoranda on the Internet to facilitate
access by Regions and delegated authorities
Regions
• develop and implement yearly training programs for permit writers
• train staff subject matter experts where necessary
• seek feedback on whether Regional staff are knowledgeable, responsive, cooperative, and
available
State/Tribal/Local
• develop yearly training programs for permit writers
• train staff subject matter experts where necessary
• seek feedback on whether their staff are knowledgeable, responsive, cooperative, and
available
We will work with representatives of permitting authorities to continually
improve permitting processes and services.
Headquarters
• identify and remove barriers to improved service
• involve regional and state permit writers in the drafting of regulations
• use formal and informal mechanisms to identify permiters' needs for guidance, training,
interpretations, new regulations, etc.
• seek customer feedback on whether Headquarters' efforts are actually improving
permitting processes and services
Regions
• involve regional permit writers in the drafting of regulations
• use formal and informal mechanisms to identify delegated programs' needs for guidance,
training, interpretations, new regulations, etc.
• identify and remove barriers to improved service
• seek customer feedback on whether the Region's efforts are actually improving permitting
processes and services
State/Tribal/Local
• identify and remove barriers to improved customer service
• participate through appropriate organizations in the development of EPA regulations
• use formal and informal mechanisms to identify the regulated community needs for
guidance, training, interpretations, new regulations, etc.
• seek customer feedback on whether efforts are actually improving permitting processes
3 -3
-------
and services
We 'will make our permit decisions within the time frames that are established
for the type of permit being requested
Headquarters
strive for timely decisions, regulations, guidance, interpretations, training, etc.
• seek customer feedback on the timeliness of decisions and services
Regions
• notify states or applicants of expected response dates to submittals and meet or beat those
timeframes; if you can't meet the timeframe, notify the state or applicant prior to the original
due date and set revised date for the Agency response
• seek customer feedback on the timeliness of decisions and services
State/Tribal/Local
• notify applicants of expected response dates to permit applications and meet or beat those
timeframes; if you can't meet the timeframe, notify the applicant prior to the original due date
and set a revised date for the Agency response
• seek customer feedback on the timeliness of decisions and services
3 -4
-------
tomer Service in ~^J Chapter 4
Feedback and Measurement
One of the basic tenets of customer service is to ask customers what they want,
or you 'II likely get it wrong. This chapter introduces the concepts and
procedures that will help your organization develop an effective system to
identify customer needs and measure satisfaction.
Why Do I Need Customer Feedback?
Your customers: ideas can help you provide better environmental and public health protection. By
obtaining feedback from your customers, you can measure progress toward your customer
satisfaction goals. Ultimately, customer feedback will help you improve the way your organization
provides products and services to its customers.
What services do permitting customers usually want? We=ve found that applicants usually seek clear,
timely, and fair permits. Communities want permits that protect their environment. Permitting
authorities rely on others to supply resources, expertise, guidance, and support in an effective manner.
Your customers may have additional, or different, needs.
How do you know if your customers are getting the services they deserve? It has long been a basic
tenet of customer service that you must ask your customers what they want, or you will likely get it
wrong. If you assume you know what your customers want, they may perceive you as arrogant or
you may spend resources on program changes that they don=t want or need. Meanwhile, you may
not notice a very simple, inexpensive, and straightforward need — because you didn't ask.
Also, asking your customers what they want promotes your agency=s image as a more customer-
focused organization. It shows you care about their opinions and that you are truly interested in
serving them better. Also, reaching out to your customers gives them a sense of ownership of the
program, generally resulting in more support for decisions reached.
You may even need to revise or expand the standards you set at the beginning of your customer
service effort. You may find out the customer service goals you initially set are too high or too low.
That is okay — feedback is, by definition, meant to be a dynamic process and the feedback loop is
a crucial element of achieving excellence in customer service.
Why Should I Set Customer Service Standards and Goals?
As with any major project, you should set goals to identify where you're going and how you'll know
when you get there. When you are seeking excellence in customer service, you'll have to initially
4- 1
-------
make some assumptions about your customers' needs so you can start asking them questions.
These assumptions will show up in your initial goals and standards. While EPA has developed four
standards for Customer Service in Permitting, as described earlier, they should not be construed as
the only right ones. Your organization should set standards that best fit your mission.
You should set quantifiable and measurable goals for each standard. These goals will help you design
the feedback and measurement system.
How Do I Design a Feedback and Measurement System?
This toolkit describes three phases for developing and implementing a successful feedback and
measurement system. (Additional phases needed to analyze and act on the feedback are presented
in Chapter 5.) You can find more detailed information in "Hearing the Voice of the Customer -
Customer Feedback and Customer Satisfaction Measurement Guidelines" OP-235-B-98-003,
November 1998. This document may also be found in the EPA web site at:
www.epa.gov/customerservice/guide.htm.
The three development and implementation phases are:
• Plan the customer feedback project
• Construct the data collection procedures
• Conduct data collection
4-2
-------
THE PLAN checklist
get ready
see what feedback you already have
decide which core questions to ask
decide frequency for customer feedback
define the target customer population
identify services supplied to customers
establish purposes of customer feedback
decide whether to contract out
develop written plan
determine resources needed
obtain agreement to proceed (if needed)
PLAN THE CUSTOMER FEEDBACK PROCEDURE
How Ready Is My Organization for Customer Feedback?
As you begin to plan, consider how ready your organization is for customer feedback by asking these
questions:
$ Do staff understand why the organization needs customer feedback?
$ Do staff members and managers sincerely intend to pay attention to customer feedback and
act on it?
$ Are key managers committed to taking action based on customer input?
4-3
-------
Preceeding Page Blank
$ Have staff members directly participated in defining the need for customer feedback and in
identifying the approaches to use for obtaining it?
$ Have managers, employees, and other users of customer feedback information expressed their
needs, issues, concerns, and objectives?
$ Is there managerial and employee buy-in and ownership?
$ Are there few or no barriers C such as concerns about change, extra work, adverse findings
C to using customer feedback successfully?
$ If there are barriers, are there identified methods to overcome them?
If you answered "yes" to these questions, your organization is clearly ready for customer feedback.
If you answered "no" to some questions, you might consider what you can do to prepare your
organization to obtain and use customer feedback.
If your organization is not fully ready for customer feedback, you should not necessarily halt your
feedback activities. Instead, just understand that you will probably face some challenges in getting
the work done, getting managers to pay attention to findings, and assuring customers that your
organization is committed to implementing the changes they want. You may need to start slowly,
collecting unsolicited feedback and using informal opportunities to gather customer input. You can
make some positive changes based on that feedback, and build a case for seeking broader and more
formal customer feedback to verify and expand the anecdotal information you gathered.
What Kinds of Customer Feedback Are Already Occurring?
Before proceeding with a new customer feedback activity, check with EPA=s Customer Service
Program in the Office of Policy to see whether any similar work has been conducted recently. You
also should check with delegated program representatives (for certain customer feedback functions).
By obtaining information from other sources, you may avoid unnecessary duplication, save time and
money, and make the best use of previously gathered data.
What are the Core Questions I Should Ask of My Customers?
We believe it is important to have some core questions that are always used by those doing customer
feedback within an organization. Core questions represent broad levels of understanding and
impressions about expectations, organizational responsiveness, and customer satisfaction. By using
core questions, we can compare and aggregate customer feedback information, both across the
agency and over time.
The following are the core questions that we endorse:
4-5
-------
*• Overall, how satisfied are you with the way the permitting process was managed?
123456
not at all very
*• How courteously did our organization staff treat you?
123456
not at all very
*• How satisfied are you with the quality and timeliness of the communications you have
received from our organization?
123456
not at all very
> How fully did our organization respond to your needs for guidance, information, or
technical support under the permit process?
123456
not at all very
How Often Should We Ask Customers for Feedback?
Many organizations contact their customers once a year to get an overall measure of satisfaction.
Other types of feedback, such as follow-up telephone calls or comment cards, provide immediate
information at the point of contact. When organizations need targeted customer information, most
find it useful to conduct multiple studies each year.
As a rule, a customer service program does not want to overburden its customers, so take care to:
$ Avoid activities that duplicate work conducted
$ Avoid contacting the same customer repeatedly
$ Seek consent from customers to participate in feedback projects, especially those that are
lengthy or where customers have been contacted previously.
There is no standard answer to how often you should ask for feedback. The frequency of customer
feedback will depend on several factors:
$ Were the findings of previous customer feedback studies positive or negative? If your
organization took action in response to concerns customers raised, has there been enough
time to see whether those actions have been effective?
$ Considering the issue(s) involved in the feedback activity, how often does it make sense to
solicit customers: opinions?
4-6
-------
$ Can you distinguish annual versus ongoing information needs?
$ Is there a way to collect feedback during your everyday customer transactions? Is there a way
to match feedback with your organization-to-customer transactions? Can you ask customers
at the end of a call if the information provided was useful? Is there any follow-up with them
later to see if they used the product provided?
$ Has some critical event occurred for which customer feedback would be important? (e.g., was
the office reorganized to speed customer service or product delivery?)
$ Do you anticipate program changes that may require surveying customers both before and
after the change?
How Long Should Feedback Activity Take?
Obviously, many variables can affect the time it takes to complete a feedback effort. These variables
include the type and method of feedback, number of respondents, and the extent to which your
organization is prepared to plan and act on the results. Many people, including the customer, will
have expectations about how long the effort will last and when results may become available.
Therefore, you should carefully plan the schedule of a feedback effort. Figure 4.1 is an example of
the timetable for one feedback survey:
Figure 4.1
CUSTOMER FEEDBACK SURVEY - PROJECT TIMETABLE
DELIVERABLE
TIME FRAME
Project Planning and Design
June
(2-3 meetings)
Design Survey Instrument
- focus groups
- internal draft of questionnaire
- 1st draft to survey team
- markup meeting
- 2nd draft to survey team
- revised draft sent to field
- final version sent for approval
- final approval from agency
6/22 & 6/29
7/7
7/10-7/12
7/12
7/17-7/20
7/25
8/4
8/7
4-7
-------
Data Collection
- field testing
- revisions (if necessary)
- phoning
Analysis and Report
- analysis
- report
- briefing charts
Process Improvement Workshops
- coordinating committee
- executive board
- notes to coordinating committee
- notes to executive board
Performance Standards and Process
Improvement Implementation
- action teams
8/14 & 8/15
8/16-8/18
8/21 -9/15
9/15-10/15
10/17
10/31
11/1 &11/2
11/8 & 11/9
11/16
11/27
start 12/10
Why Should I Establish Quality Control Procedures?
Developing and applying good internal control procedures helps ensure the quality, reliability, and
integrity of information used for decision making. You should apply quality control standards and
techniques to data collection, analysis, and reporting of results.
Controls may be as simple as limiting access to raw, customer-specific data, as well as separating the
data collection, administrative and presentation duties from the affected action officials.
Alternatively, controls can be as thorough as performing independent quality assurance reviews.
Internal controls should provide reasonable assurance that your customer feedback objectives will
be accomplished reliably and cost-effectively. See Appendix A for a description of specific control
standards and techniques that apply to EPA.
Figure 4.2
4-8
-------
Checklist
Establishing the Purposes of Customer Feedback
Define the feedback objectives
What do I want to accomplish with this feedback?
Determine how the findings will be used
What will we do with the findings?
Will they be used:
- as a key business performance indicator?
- to revise, correct, or improve a process?
Checklist - continued
- to identify customer needs and expectations?
- as a management tool for customer relationships?
- to inform planning, decision making, and resource allocation?
- to reward, recognize, or compensate employees?
- to help validate standards, specifications, and measures?
Determine who will use the findings
Who else is interested in the findings?
How much time are they able to give to the process?
How would they prefer to learn about the findings - in briefings, written reports, graphics,
action plans, e-mail?
Figure 4.3
4-9
-------
Checklist
Developing a Written Plan for the Customer Survey
purposes of the activity
quality control procedures
ways findings will be used
identification of target group
methods of data collection
timing for data collection
analysis plan (see Chapter 5)
tools for carrying out feedback activity
• discussion topics
• survey instrument
• database
anticipated products
• tables and graphs
• interpretive text
• slides
• specific conclusions
• recommended actions
CONSTRUCT DATA COLLECTION PROCEDURES
What Is the "Best" Approach for Assessing Customer Satisfaction?
Your best approach will depend on the kind of product or service you provided, the kinds of
customers you served, how many are served, the longevity and frequency of customer-supplier
interactions, and what you intend to do with the results. Two very different approaches can produce
useful findings:
$ Continuous assessment methods C methods to obtain feedback from the individual customer at
the time of product or service delivery (or shortly afterwards).
$ Periodic survey approaches C methods that obtain feedback from groups of customers at periodic
intervals after service or product delivery. They provide an occasional snapshot of experiences
and expectations.
You need multiple inputs from customers to understand their expectations and satisfaction. It is like
peeling away layers of an onion - each layer reveals yet another deeper layer. Both approaches can
help you assess your organizations overall accomplishments and areas for improvement.
4- 10
-------
Continuous Assessment
While this Toolkit focuses on methods for obtaining customer feedback periodically, you also can
adopt continuous assessment as a standard method for obtaining customer satisfaction information.
Include continuous assessment by:
$ enclosing a feedback card with every draft and/or final permit issued
$ enclosing a feedback card with every public comment response mailed
$ making a follow-up phone call to every customer (or to every fifth, or twelfth, or nth
customer) within one or two days of contact
Decide on Data Collection Method
Informal methods for obtaining information from customers clearly produce valuable information.
Everyone needs to use these everyday opportunities for customer feedback. Use this information to
complement the more systematic forms of gathering feedback discussed earlier.
Formal methods frequently used to gather customer feedback include focus groups; mail-back
postcards included with materials you send to your customers; mail surveys; telephone surveys;
publication evaluation forms; printed or in-person surveys (possibly including computer-assisted
personal interviews or an intercept survey through which you ask every nth customer attending a
function or visiting a facility to participate). Electronic mail will become an important means for
collecting customer feedback as more people gain access to the Internet.
When picking a method, you should consider several factors, such as the types and number of
questions you will ask. Your decision also will be affected by available resources, how fast decision
makers need to have the information, and how representative the findings need to be. The response
rate — the number of customers who actually answer questions divided by the number contacted for
information — is also an important consideration because it will affect the way you can use findings.
A summary of different methods appears in the table on page 4-12.
If you choose a mail or phone survey, you will need an accurate name, address and/or telephone
number. At times you=ll need to know which EPA programs or services the customer sought or
received, as well as any demographic information available.
Note that several different practices can affect the ratings of various data collection methods:
$ Focus groups, telephone and in-person surveys require trained staff to conduct proper interviews
and prevent interviewer bias. They can also demonstrate through direct personal contact that the
agency takes customer feedback seriously.
$ Telephone surveys can more readily accommodate differences in language and literacy levels than
can mail surveys, but they cannot accommodate lengthy questionnaires or visuals. However, some
4- 11
-------
people do not have a telephone, and many who do will refuse to participate in telephone
interviews.
$ Mail surveys can be longer, since respondents can work at their own pace, but they have the
longest response time and may not reach the intended target.
$ Mail surveys allow no interviewer bias to creep in, but they offer little ability to probe or ask
complex questions, and may generate ambiguous answers.
$ Your advance and follow-up efforts can dramatically improve costs, timeliness, and your ability
to generalize results. Mail surveys can follow up with customers who do not respond initially.
An advance letter can increase participation and response rates for mail and telephone surveys.
Such letters also allay customers' concerns about such matters as: how they were selected, why
they have been selected to participate again (if applicable), anonymity, how long it will take them
to answer the questions, and how findings will be used. (See the sample advance letter on the
following page.)
4- 12
-------
[on your organization's letterhead]
Mr. John Doe
Alpha, Beta, and Gamma Co., Inc.
555 Main Street
Anywhere, USA 12345
Dear Mr. Doe:
I am writing to let you know that your name has been selected at random to participate in a survey about
business owners' experiences with our agency. You are one of a small group of people we are
contacting. Your feedback can help shape our future direction.
We at [XXX Agency] will take findings from the survey into consideration as we develop our plans for
the next decade. We are committed to incorporating customer viewpoints and recommendations into
our
strategic planning, budgeting, and decision-making while recognizing the need for balancing sometimes
competing and conflicting interests.
I realize that we may have contacted you before to answer similar questions. We are tracking our efforts
to respond to customer concerns, so it is very important to hear from you again. You will not be
identified personally.
You should receive the survey in the next few days. It will take less than 10 minutes for you to
complete.
I urge you to consider the questions carefully and let us know how we can better serve you. Meanwhile,
if you have any questions, please call 1-800-xxx-xxxx to speak with a staff member on XXX Agency's
survey team (or someone at YYY Consulting, the firm conducting the survey for us).
I thank you in advance for your time and consideration.
Sincerely,
Name
Title of highest possible agency person
4- 13
-------
Comparison of Feedback Methods
Factor
Cost
Convenience for customer to
complete
Length of survey
Ability to encourage customer to
participate
Ability to provide instructions or
explanation to customer
Requires customer to initiate
Respondent's perception of
anonymity
Types of questions
Opportunity to probe and ask "why"
questions
Need for accurate list of telephone
numbers or addresses
Allows "branching" and skip
patterns***
Ability to get quick response
Response rates
Extent of likely bias between
customers who choose to participate
and those who decline
Ease of data entry
Extent of data clean-up
Ability to generalize results
Focus Groups
moderate
moderate
up to 2 hours
high
high
no
moderate
closed- and
open-ended
very high
no
yes
moderate
high
high
moderate
low
low
Mail-back
Form*
low
high
very short
low
low
yes
high
yes/no
limited
no
no
no
low
high
moderate
some
low
Mail-out Survey
moderate
high
up to 12 pages
moderate
low
no
moderate
mostly closed-
ended
limited
yes
some
moderate
moderate to high
some
moderate
moderate
high
Telephone Survey
moderate
moderate
12-15 minutes
high
high
no
moderate
closed- and open-
ended
moderate to high
yes
yes
yes
high
low
high
low
high
In-person Survey III Continuous III Electronic via
III (Every nth III Internet
HI Customer)** |||
high
moderate
up to 1-1/2 hours
high
high
no
moderate
closed- and open-
ended
high
yes
some
moderate
high
low
moderate
moderate
moderate to high
moderate
moderate
mixed
mixed
mixed
mixed
moderate
most closed-
ended
limited
yes
some
moderate
moderate
mixed
mixed
mixed
high
very short
low
low
yes
not anonymous
can be both
limited
no
no
high
low
high
low
low
Refers to feedback forms or postcards distributed at a point of contact with the customers or included among materials sent to a customer.
Every nth customer would be contacted using telephone, feedback card or survey form ; therefore "mixed" appears frequently in the table.
"Branching" is a particular answer to one question that leads to a series of related questions. For example, if a customer cited a product as especially helpful, a series of questions might try to
ascertain what, exactly, about that product led to the customer's opinion; if another customer finds a product especially irrelevant, a different series of questions might be posed. "Skip patterns"
refer to instructions about which questions to answer and which to skip, depending on the particular answer given to one question.
The Sample
4- 14
-------
If the number of customers of interest is relatively small, not more than 50, you could contact each
customer to obtain feedback. This is the census approach. In many cases, you provide services or
products to a large group of customers, perhaps too large for a census approach. In such cases, a
sampling approach is needed, and two options are possible: (1) & judgment sample, in which you
consciously select the customers that you will contact from the entire group served, or (2) a
probabilistic sample, in which customers are picked randomly from the entire group served during
the period of interest (e.g., the past year).
In most cases, it is better to rely on the probabilistic approach. Judgment samples may be biased
because of the way customers are selected. If a sample is biased, it is impossible to draw inferences
about the entire group of customers served. As long as the response rate is high enough, probabilistic
samples are not biased, so inferences can be made about the group as a whole.
Determining the Sample Size
If you choose to conduct a mail, telephone, or in-person survey, you will need to decide the number
of people selected to participate. To determine this number — the sample size — you should
consider several factors, such as the total number of customers served, the intended use of the results,
available resources, and time.
The larger the percentage sampled, the more certain you can be that the feedback represents the
results you would have obtained if you had surveyed every customer. The smaller the percentage
sampled, the greater the likelihood that feedback will differ significantly from those in the master list.
The relationship between sample size and accuracy of findings is due to sampling error, which
indicates the extent to which the sample is different from the entire group under study. In a news
article that reports the President's approval rating as 62 percent, plus or minus 5 percent, the "plus-
or-minus" value is the sampling error.
To decide the size of the sample, you can either:
Determine the largest sample size that you can afford and calculate the associated sampling
error, or
• Determine the maximum sampling error that is acceptable and then select the sample size that
will produce that level of error.
The sampling error can be estimated through a confidence interval, which specifies a range of values
within which the true measure is found. Typically, survey results rely on a 95 percent confidence
interval, but lower levels are acceptable, depending on how you plan to use the findings. Popular
media reports rarely stipulate confidence intervals, but they are implied. Again using the President's
popularity rating as an example, the unstated premise is that the analyst is 95 percent certain that the
President's popularity is between 57 and 67 percent; that is, 62 percent, plus or minus 5 percentage
4- 15
-------
points.
One last thing to consider in determining the sample size is the kinds of comparisons you will make
with survey findings. Many times, analysts are interested in comparing ways that different customers
react to various services. These comparisons may involve large vs. small businesses, the general
public vs. educators, and so forth. If these comparisons are a critical portion of the analysis, you must
plan for them in the sample design so that enough of each customer type is surveyed to make the
findings meaningful. See Appendix B for additional information on sampling considerations.
Develop the Questions
Checklist for effective questions:
use short, value-neutral statements or questions
_use simple words
avoid jargon
_be clear and easy to understand
arrange questions in logical order
_use appropriate response choices (include all possible answers and
minimize overlap among the answers)
do not use double negatives
be upbeat and interesting
_write to the appropriate reading level (9th grade or less for general
public; several word processing software packages incorporate a
feature that determines reading level)
_ use questions pretested in other surveys whenever possible
leave out the "nice to know" program/product/service questions not
vital to success
In choosing questions, you should keep two principles in mind: (1) make sure that the questions and
answers address your objectives and (2) set limits on the length of the survey instrument.
Many sources are available to help develop questions for surveys. These include software packages
such as Corporate Pulse (which is available to EPA staff through the Customer Service Program),
prior surveys sponsored by EPA and other agencies, journal articles, and item banks maintained by
some universities and survey organizations. When possible, it is better to use a previously tested and
validated question, rather than one newly created for the current survey.
Survey questions are generally of two types: open-ended and closed-ended. In open-ended
questions, the customer creates his or her own answers. The following are examples of open-ended
questions:
• Do you have any suggestions for improving service? [IF YES], What are they?
4- 16
-------
• How could our organization be more responsive to your concerns?
• Could you please describe the most satisfying experience you 've had with our organization ?
Closed-ended questions limit the responses a customer can provide. They may include yes/no
answers, categories of responses, rank-ordered responses, or scales. The following are examples
of each type:
Yes/no
In the past 6 months, have you contacted the XYZ office?
1) yes 2) no
Categories
In what kind of community is your business located? Would you say it's. ..
1) urban 2) suburban 3) rural
Rank order
Of the following items, which 3 are most important to you? Please indicate with a "1" for the most
important, a "2" for the next most important, and a "3" for the third most important.
clean air
clean water
hazardous waste disposal
a minimum level of government regulation
lower taxes
Scale
Please rate your satisfaction with the service you received, using a scale of 1 to 6. "6" means
you are very satisfied, and "1" means you are very dissatisfied.
123456
With closed-ended questions, it is relatively easy to record and analyze responses, and you will not
receive irrelevant or unintelligible responses. However, you risk "missing the boat." To illustrate,
suppose you ask the closed-ended questions, "What was the main reason for your visit?," giving
several possible answers, and 30 percent of your respondents mark "other." Drawing valid
conclusions about why customers visited would be hard. If you decide to use closed-ended questions,
pretest them to identify all the likeliest responses to your questions.
In developing closed-ended questions, you should carefully consider the advisability of including
response options such as "don't know" and "no opinion." While customers should not be forced into
providing responses when they really do not have answers, it is better to find ways to encourage a
response than to let customers default to a neutral position. In mail surveys, you can provide this
4- 17
-------
encouragement through instructions; in telephone and in-person surveys, you can encourage
responses by not offering "don't know" and "no opinion" as response options.
On the other hand, including "not applicable" as a response is important in mail surveys, so that
customers are able to indicate this when they have not had a particular experience. In asking
questions about a past event, consider giving a "don't remember" option. Keep the survey to a
reasonable length by asking only the questions needed to address the issues that prompted your
survey; leave out the "nice to know" questions.
Recognize that open-ended questions will provide a richness of data that can complicate analysis.
Reducing responses to a few categories that can be coded, entered into a data base, and analyzed can
be difficult. It is probably best to use a mix of questions, both closed and open, in most customer
feedback questionnaires.
If you are planning an ongoing or periodically repeated survey, identify a few key program goals that
are unlikely to change very soon, and focus your questions on them. Develop questions that will
indicate how well customers think the goals are being met. These key questions need not be elaborate
or profound, but should be very basic. To effectively compare results over time, you need to use
essentially the same core questions in your survey on each iteration. You will need to avoid making
any major changes to these key questions, whether in wording, scaling, or placement, so be sure to
ask the right questions from the beginning.
Make sure your questions are relevant to your customers. Although this may seem obvious, it is
important to remember. Be particularly wary of questions that may be interesting to ask, but may
only add time and cost while not producing useful information. These questions could be:
Extraneous questions that do not address the stipulated purposes and objectives of the
feedback activity.
• Questions that are subj ect to misinterpretation. These may have vague words, use unfamiliar
jargon, or could be understood differently by different types of customers.
Double-barreled questions that embed more than one item, such as "On a scale of 1 to 6,
please indicate how clear and useful the materials are." The customer may have one opinion
about clarity and another about usefulness, but is not given an opportunity to distinguish
between them.
Questions that may upset some respondents. Questions that may seem intrusive, such as
household income, are best when worded neutrally (such as by asking whether the customer's
household income falls above or below a certain level) and placed at the end of the survey.
Questions on potentially sensitive or offensive matters, especially about cultural, ethnic,
gender, and socioeconomic considerations.
4- 18
-------
• Questions that do not elicit responses pointing to specific remedies.
If you don't ask the right questions in the right way, relatively soon after the service experience,
feedback will not be as useful as it might have been. Also, remember that to compare results over
time you should avoid making major changes to key questions, whether in wording, scale, or order
in the questionnaire.
There is no single correct scale to use. However:
• Whenever possible, the same scale should be used throughout a given questionnaire to help
ensure that different responses within a questionnaire can be compared validly.
Different survey efforts within an organization should use the same scale. To this end, for
national consistency, we would recommend that when using the core questions described
above, you consistently use the same scale of one to six (1 - 6), whenever possible.
Construct the Questionnaire
No matter what method you use to collect data, all questionnaires follow a similar format:
Introduction — sets forth the purpose of the survey and guides the customer through the
questions
• Experience — establishes the customer's ability to answer various parts of the questionnaire
• Measurement — asks the person surveyed to characterize his or her experiences, needs and
desires as your customer
Customer information — gathers data that will be used to classify respondents
Survey questions should be presented in a logical sequence. Many survey experts believe that the first
question on the survey, more than any other, will determine whether your customer completes or
discards the questionnaire. Starting with a fairly simple question is a good idea because it suggests
that completing the survey will be neither difficult nor time-consuming. It is also advisable to ask a
fairly interesting question to evoke interest.
The next set of questions should focus on matters that the customer is most likely to judge as useful
or salient. This continues the process of drawing the customer in so that he or she becomes engaged
with thinking about the questions being asked and becomes invested in completing the survey.
Grouping questions together that share common themes makes sense because the customer then
focuses on that area of inquiry. Group questions with similar response options. For example,
questions that have yes/no responses should be together and questions that have scale responses
should be together.
4- 19
-------
The order of questions should also mirror the thought processes that customers are likely to follow.
For example, questions about experiences with on-site inspections should precede questions about
suggestions to improve those inspections.
The final set of questions should center on those most likely to be sensitive or offensive. These may
include questions about personal characteristics (race, age, income) and unsuitable behaviors.
The final page of the booklet should not have any survey questions. Instead, it should invite the
customer's comments or suggestions about anything raised in the survey or other issues and concerns.
It should also indicate the address for returning the questionnaire (in case the survey gets separated
from the reply envelope) and, when possible, a toll-free number set up exclusively to receive survey
inquiries.
Mail Surveys. The mail survey has to do everything you would do if you were with the customer.
It has to be visually appealing, have a pleasant tone, and be clear while being concise and to the point.
The survey instrument is under the direct control of the customer. Its physical look will affect the
customer's willingness to respond; the clarity of the instructions and questions will affect the
customer's ability to interpret their meaning.
Single page questionnaires and comment cards should be attractive and easy to read. Longer
questionnaires should be printed in booklet form, on 11" x 17" paper that is folded in half and stapled
in the middle to produce a standard 8 1A" x 11" page. The cover should be visually appealing and use
a logo or other graphic design to interest the customer; no questions should appear on the cover. Use
of color ink and high-quality paper will add only minor costs to the survey, but can substantially
improve response rates and reduce the cost of follow-up correspondence and telephoning by staffer
contractors. The cover should give the title of the survey activity and indicate who is conducting it.
Note the Social Security Administration uses brightly colored paper, desk-top publishing to allow
more flexibility in design, and larger print to accommodate the needs of its elderly and disabled
customers.
The methods used to construct the questionnaire are different, depending on the mode of data
collection. In the next section we present methods for constructing questionnaires for focus groups,
mail surveys, and telephone surveys — the most frequently used forms of data collection in periodic
surveys.
Focus Groups. As knowledge about customer surveys has expanded and entered the public domain,
more and more people claim to be conducting "focus groups." It is important to distinguish between
focus groups based on scientific procedures and understanding of human interactions and casual
discussions among people who share a common interest or concern. Both approaches provide
potentially useful information, but analysts should recognize the difference.
The key instrument for a focus group is the Moderator's Guide. This is a series of questions, probes,
and discussion topics arrayed in a logical order. The moderator uses the Guide to elicit opinions and
4-20
-------
experiences from participants, and to ensure that discussions stay focused as much as possible on the
critical issues.
Typically, a Moderator's Guide is organized as follows:
Q Introductions by moderator and participants
Q Review of ground rules, such as
• You have been asked here to offer your views and opinions; everyone's participation is
important; the conversation does not need to flow through the moderator, although the
moderator will manage the group
• Speak one at a time (avoid side conversations)
• Note video taping, audio taping, and observers (as applicable)
• There are no right or wrong answers; consensus not required
• Okay to be critical; if you don't like something, say so
• All answers are confidential, so speak your mind
Q Brief explanation of the focus group purpose and introduction of the topic
Q Definitions
Q Questions, probes, discussion topics
Q Closing and thanks
Telephone Surveys. Because customers have no questionnaire in front of them during a telephone
survey, visual appeal is not an issue. However, ordering, clarity, and conciseness of questions are
important. It's also important to place the call at appropriate times (e.g., not at dinner time).
Additionally, the interviewer acts as an intermediary between the customer and the questions posed.
With this in mind, the following principles apply to telephone surveys:
The introduction the customer hears will probably determine whether the customer hangs up.
The introduction should be concise, state the purpose of the call, estimate its length, and assure
confidentiality. This is a sample:
Hello, my name is [fill in], and I'm with the XXX Agency [or 777 Consulting]. We're
conducting a survey of people who have received materials from the XXX Agency to learn
about their experiences and opinions. Let me assure you that this is not a sales call, and
that we will keep all information about you and your responses private. We will use the
4-21
-------
information you provide only to help improve XXX Agency's services. The survey will take
less than 15 minutes to complete and is purely voluntary. Is this a convenient time, or
should 1 call you back later?
Because customers will rely on verbal cues and instructions, rather than written ones, questions
should have a limited number of responses (about three or four).
Each question should be relatively short.
Avoid questions that ask the customer to look up information or check with others.
Avoid use of leading questions.
Be sure to read the questions aloud to others to see if they make sense. Remember, what works
for the written word does not always work for the spoken.
Complex skip patterns and branching are easily accommodated through computer-assisted
telephone interviewing (CATI) systems. Skip patterns occur when a particular answer to one
question means the respondent is not asked certain questions that would otherwise follow;
branching occurs when a particular answer to one question leads to a series of questions
customized to that particular answer.
Rank-order questions are subject to error in telephone interviews in a way that they are not for
mail or in-person surveys. Rather than asking a customer to rank-order a list of, say, eight items,
it is better to ask that person questions in a series of pairs ("Which is more important to you, X
or Y?"). Alternatively, you can break up the list into a series of separate scaled items ("On a scale
of 1 to 6, where 1 is extremely important and 6 is not at all important, how do you feel about
X?")
When changing subjects, telephone surveys should cue the customer with transitional language.
To accomplish this shift, use statements such as, "Now, I'd like to turn to your experiences with
• Instructions for the interviewer must be perfectly clear, and the same format should be used
throughout the survey. For example, interviewer instructions are typically enclosed by brackets,
in all capital letters.
• For a sizable telephone survey (of say, more than 50 people), use of computer-assisted telephone
interviewing (CATI) should be considered. For large studies, CATI will be more cost-effective
and produce more reliable information.
Pretest
A pretest is a small-scale trial of the instrument and data collection methods. Conducting a pretest
is extremely important because the results will allow you to refine the instrument and methods before
4-22
-------
the comprehensive data collection activity begins. It may seem that a pretest is unnecessary if a
survey has been carefully researched and designed. However, even the best plans cannot anticipate
all real-world circumstances.
Results from a pretest can tell the analyst:
- whether the flow of questions is logical and orderly
- whether questions seem relevant and appropriate to the customers
- if customers were able to easily understand and respond to questions
- if response categories are adequate
- whether questions truly reflect the issue that is intended to be measured
A pretest is helpful for cost projections, and also provides information about actual burden (that is,
the amount of time to complete the survey). This information is essential for Office of Management
& Budget (OMB) clearance (required for federal agencies, their contractors and cooperative
agreement partners performing surveys of direct benefit to the sponsoring agency). A pretest that
includes more than nine people who are not federal employees also requires OMB clearance.
One of the best ways to conduct a pretest is to randomly select individuals from the target group,
have them complete the survey, and then conduct a focus group session to review their opinions. If,
for example, you intend to conduct a telephone survey, you should recruit customers, bring them to
a central location where they can be interviewed by telephone, then meet as a group to go over the
draft questionnaire and their experiences in answering the questions. Pretest participants should not
be selected for the actual survey.
Contingency for Non-Response
Occasionally, regardless of planning, there will be times when response rates are simply too low for
you to make inferences and recommend action. In these cases, it is important to have a contingency
plan for non-response. The plan will need to include the additional steps you may need to boost the
level of responses. Some potential steps:
• Reminder Calls or Postcards — If you did not include these steps in the original survey plan, you
should consider them if the response is low. If you did include them in the original plan, it may
be advantageous for you to repeat them.
• Follow-Up Contact with Non-Respondents — You may need to make telephone calls or other
types of personal contact to non-respondents to identify the reasons for their non-response. Find
out if they understood the intent of the survey and the questions, if the questions were relevant
and if there were specific factors that persuaded them not to respond.
• Improve Contact Information — It may be that many addresses or phone numbers of the target
4-23
-------
group are incorrect or out-of-date. Updating this information would very likely improve the
response rate. Places to check include the Internet, credit bureaus, and business directories.
• Revise Survey Instrument — Some questions may make respondents feel uncomfortable, so you
may need to revise the instrument. NOTE: If you change the survey instrument significantly, you
may not be able to compare the results received before the change with those received after the
change. You will need to carefully consider the trade off of response rate vs. data validity.
Some of these steps may require a great deal of effort, time, and money. The group or individual in
charge of the survey will need to carefully consider the various options. If the response rate remains
too low, you may need to wait for a better time and customer base, or to rely on direct conversations
with customers.
OMB Clearance (EPA Only)
Under the Paperwork Reduction Act of 1995, the U.S. Office of Management and Budget must
approve any federally-sponsored collection of information that asks the same question of more than
nine non-federal respondents. Typically referred to as "OMB Clearance," the process is an exacting
one and demands strict adherence to OMB requirements. For example, if a customer feedback
activity is subject to OMB clearance, the cover of the data collection instrument must contain
standard language and the date on which the clearance expires.
EPA has obtained OMB approval of a generic Information Collection Request (ICR) to conduct
customer satisfaction work. Under this authority, the clearance process is streamlined and the time
for clearance is reduced from as long as 6 months to between 10 and 15 days. This generic ICR is
available only for strictly voluntary collections of opinions from customers who have experience with
the existing product or service.
Appendix C explains the streamlined process and provides several examples. You may request the
fact sheet as a separate electronic document from Patricia Bonner, Director of EPA's Customer
Service Program (Mail Code 2161). EPA offices may also send her survey instruments for quick
review to ensure that questions are properly worded to address customer satisfaction issues. In some
cases, another information collection request may be more appropriate to use than the generic
clearance mechanism.
Proposed EPA survey packages should be sent for final review to Barbara Willis of the Regulatory
Information Division at EPA Headquarters (Mail Code 2137). She will check the package for
compliance with OMB regulations regarding the generic clearance and review the burden placed on
the public, state officials, tribes, and other non-federal government customers. She will forward to
OMB all survey instruments and the required clearance package.
See Appendix C for more information about specific procedures to follow, forms to complete, and
general information about EPA Customer Feedback OMB Clearance. EPA personnel listed below
may be able to provide additional information:
4-24
-------
Barbara Willis
202-260-9453
202-260-9322 (fax)
willis.barbara@epamail.epa.gov
Model Survey Instruments
Pat Bonner
202-260-0599
202-260-4968 (fax)
bonner.patricia@epamail.epa.gov
Appendix D contains three model surveys prepared by the workgroup that you may use as is, or
modify as desired. The three customer groups targeted are the permit applicants, citizens, and the
delegated authorities. Please note, while these are similar to previously developed permitting surveys
with OMB clearance, those approvals have expired. EPA offices must re-obtain OMB clearance to
use them. These approvals would be available under the above described streamlined process.
Additional Resources
The EPA Customer Service Program collects copies of survey instruments, reports, and resulting
plans. These materials are a resource for other EPA offices and staff who want to learn more about
their customers.
4-25
-------
CONDUCT DATA COLLECTION
No matter what data collection methods you choose, you need adequate planning, training, quality
control, and supervisory practices to ensure that the information collected is:
• timely
• accurate
• efficient
• parsimonious
• reliable
• valid
cogent
Focus Groups
A focus group project typically involves several steps, as discussed below.
To recruit participants, you will need to compose an effective recruitment script. Use this tool to
create dialogue between the person recruiting participants and the candidate, and to qualify potential
participants, considering factors such as age, socioeconomic status, and race/ethnicity. Then you will
invite individuals who meet requirements to participate in the group. You should recruit about twelve
qualified participants for each focus group. Allowing for last-minute change of plans and illness, the
moderator should expect nine will attend.
Several practices can maximize the efficiency of the recruitment process:
• Well before the group meets, mail a letter to participants that confirms the date, time, and location
of the group and states whether the respondents will be paid for participating. The letter thanks
the participants, gives directions to the focus group facility (including a map) and repeats the
general objectives of the group.
• Also, you may decide to provide transportation to the focus group facility for those who need
it.
• On the day of the focus group (or the previous day, if the group is scheduled for the morning),
make & follow-up telephone call to the participants to remind them to attend.
Running a successful focus group also requires:
arranging for focus group facilities
• providing video and audio taping equipment or people assigned as recorders
• providing a video hookup between the room where the focus group will meet and the room where
you (or others) will observe the focus group (if this is part of the design)
coordinating participants' schedules
4-26
-------
During the focus group, it is a good idea to use both a moderator and an assistant to conduct the
session. The moderator will pose questions to elicit candid opinions from the participants, keep the
discussion moving, cover all topics in the discussion guide, recognize when participants bring up
valuable new information, and steer the discussion in that direction if warranted. The assistant
supports the moderator as needed, takes notes, and handles logistics.
Mail Surveys
In setting up data collection procedures for a mail survey, a good database is important. The
database should contain, for each customer, a unique identification number, the customer's
characteristics relevant for the sample selection (such as geographic location, size of business, or date
of last contact with your organization), name and address, mail-out date(s), and the date the response
is received. This database is a tracking system.
A mail survey typically includes several mailings, each of which experts call a "wave." Send out each
wave on the same date:
• If you use an advance letter, mail all of them to customers on the same day.
• About a week later, mail the first questionnaire to all customers. Attach a label with the unique
identification number to each questionnaire. A cover letter should refer to the advance letter, ask
for cooperation, and (when possible) provide a toll-free number for customers with questions.
The package should also contain a prepaid, pre-addressed envelope for returning the completed
survey.
• As completed questionnaires come in, record them in the tracking system. As undeliverable
questionnaires come back (e.g., the customer has moved and left no forwarding address or the
address is incorrect), note this in the tracking system.
• About three weeks after mailing the first questionnaire, send out the second copy to all those who
have not yet responded. A cover letter should note the importance of the study and ask
customers to respond. The second copy of the questionnaire should be a different color from the
first version. This distinguishes between the two copies, sends a signal to customers, and aids
efforts to track responses.
The following often help improve response rates:
• the advance letter (if used) should be on official letterhead, with a signature or title that is
meaningful to the customer
any signed correspondence should use a real signature, rather than a rubber stamp (scanning in
the signature can work well for many letters)
4-27
-------
• use a "live" stamp (if possible), rather than metered or prepaid postage, to send out the survey
• use "address correction requested" to get information on customers whose surveys cannot be
delivered, then use the corrected information in the next mail-out
• use a large enough envelope so that the survey booklet does not have to be folded
establish, when possible, a toll-free number for the duration of the data collection period, and
encourage customers to call with questions or comments
• allow respondents to fax back the completed survey
• if the budget permits, send out a third mailing via certified mail or using an overnight delivery
service (this is a last resort and may produce only minimal results)
Data from mail surveys must be key-entered or scanned. It is usually most cost-efficient to wait until
you have a sizable batch of completed surveys before beginning data entry procedures. Be sure to
do a periodic quality check to uncover data entry errors.
Telephone Surveys
Whether using computer-assisted telephone interviewing technology (CATI) or a traditional paper-
based technique, you must train telephone interviewers specifically on the study's questionnaire and
data collection procedures. You should cover the following topics during interviewer training
sessions:
• Background and scope of the survey. A project leader gives interviewers general information
about the background and scope of the project. She/he explains the types of information to be
collected and how that information will be used.
• Review of the questionnaire. A person responsible for data collection goes through the
questionnaire and leads an item-by-item discussion.
• Dealing with uncooperative respondents. Experienced staff lead discussions about ways to
start off the interview right, enlist cooperation, build rapport, and minimize break-offs and non-
responses. The interviewers will also review strategies for ways to manage challenging situations.
• Answering customers' questions. Some frequent questions:
> How was I selected?
*• What is the survey about?
> Who is conducting it?
> Who wants to know these answers?
> How will the information be used?
> How long will this take?
4-28
-------
•• Will I be identified?
> How do I know you are not some con?
• Quality control procedures. Project leaders monitor such matters as posing questions
accurately, tone, courtesy, responsiveness to customers' concerns throughout the survey, and
reviewing these procedures with interviewers. Telephone interviews for any sizable study are
usually conducted using Computer-Assisted Telephone Interviewing (CATI) technology. CATI
systems:
> greatly reduce the possibility of mistakes
> ensure accurate recording of the survey response
> instantly establish a tracking system and a record of each call
> provide significant improvements in quality control and efficiency, and allow complex
branching and skip patterns
Electronic Feedback
As access to the Internet spreads, electronic communication will become an important method for
gathering customer input. You can easily collect feedback by asking Web page visitors a few
questions, inviting grantees to complete comment forms and submit them electronically, and through
on-line discussions in "chat rooms."
E-mail surveys are one of the fastest and least intrusive means for gathering customer feedback. Up
to 50 percent of the responses are received within 24 hours. They are also cheaper to conduct since
you pay no interviewers, or printing and distribution costs. In addition, since e-mail is generally not
routed through others, the survey will get to the intended individual. However, respondents are not
anonymous.
4-29
-------
Chapter 5
'iS^vin Responding to Feedback
This chapter offers methods to analyze and present the data received under
a feedback and measurement effort so that meaningful results can be
determined and acted upon.
Now that you've collected customer feedback, you need to understand the data and respond
appropriately to what the data is telling you. So:
•_ ANALYZE the data
^ ACT on the results
5- 1
-------
ANALYZE THE DATA
Figure 4.2 in Chapter 4 described the steps you should take for establishing the purposes of customer
feedback. These steps included preparing an analysis plan and identifying the products your feedback
proj ect would produce. However, you can modify your framework for analyzing findings at any time.
Your analysis plan should specify how your organization will analyze the survey responses to produce
the desired products. This plan ensures that your data will answer the overarching questions being
posed, and that you do not gather extraneous data. It also sets expectations about the kinds of
information that will result from customer feedback.
Your analysis plan should: (1) designate the dependent and independent variables and (2) identify
the unit of analysis. Dependent variables are the phenomena you are investigating. In this case, the
dependent variable probably will be the degree of customer satisfaction with a specific product or
service. Independent variables help explain the dependent variable data you collect. For example,
they may include differences in the product or service provided (e.g., customers were consistently
more satisfied with one service than with another), variety in the frequency and type of interaction,
or differences among customers (e.g., educators, students, local planners and small business owners).
The unit of analysis is what you are studying. In customer feedback surveys, the unit of analysis will,
in most cases, be the individual person served. When you use continuous feedback methods, the unit
of analysis will generally be the individual customer transaction.
Data Clean-up
Once you have set up the database and entered all data, you must review the data and prepare it for
analysis. This may involve several activities, such as deleting cases that left all answers blank on a
mail survey and coding open-ended responses into categories. Generally, this is the time to run a set
of frequencies to show the number of yes and no responses to each question and the total number of
responses of all kinds to each question. This quick analysis gives you a rough check on the
completeness and accuracy of your data (the total number of responses to any one question cannot
exceed the total number of respondents and will rarely differ greatly from the total responses for each
of the other questions). Frequencies flag out-of-range values (i.e., responses to one question that are
so different from responses to similar questions that you doubt their accuracy).
Types of Data and Analyses
Data from focus groups tend to be qualitative in nature. Analysts may tabulate data from focus
groups, such as "X percent of the participants expressed satisfaction." You should treat these
numbers cautiously and not generalize them to the full set of customers because focus groups usually
have a small number of participants who may have been recruited because they had specific
experiences or characteristics. You may review transcripts from focus groups to detect patterns and
inconsistencies or you may apply more rigorous content analysis.
For mail and telephone surveys, you can produce a variety of statistics:
5-2
-------
• Descriptions of central tendencies, such as the mean (the average value), median (the middle value
- half are larger and half are smaller), or mode (the most frequently occurring value).
Other descriptive statistics, such as frequencies, percentiles, and percentages. In customer
satisfaction surveys, the most commonly reported result is the percentage of respondents who
expressed satisfaction with a specific aspect of their interaction with your organization.
Cross-tabulations that array independent variables against the dependent variable (for example,
type of customer displayed against a summary measure of customer satisfaction, such as the
percentage of customers of each type who reported being satisfied with the product or service
they received).
• Multi-variate statistics—such as factor analysis, analysis of variance, and regression analysis—to
determine the relationship between and among selected variables.
Chi-square, z scores, t-tests, and other statistics to determine statistical significance.
Time-series and trend analyses to determine long-term changes, seasonal, and cyclical patterns
in the data.
In most cases, focusing on bullet items 2 and 3 above will meet all your needs and expectations.
Analysis: An Example
The following example demonstrates how you might analyze data from customer feedback. Suppose
your organization has distributed several thousand copies of the ABC Booklet, and you asked a
sample of 450 customers this question:
On a scale ofl to 6 where 1 represents "highly dissatisfied" and 6 represents "highly satisfied, "
how would you rate your satisfaction with the ABC booklet you receivedfrom our organization ?
The most straightforward way to analyze the responses is to provide the average score, which in this
case is 3.5. Although an average score is a very important piece of information, you can do a lot
more with the data from your customers. You might begin with a frequency distribution, where you
determine the number and percentage of respondents who gave each score between 1 and 6. Here
is one way to present that distribution:
5-3
-------
Customer Satisfaction with the ABC Booklet (n = 450)
Score
1 — highly dissatisfied
2
3
4
5
6 — highly satisfied
Total
Number
42
27
122
132
38
32
393
Percent of those
expressing an opinion
11
7
31
34
9
8
100
don't remember receiving the ABC Booklet: 22 (5 percent of 450)
don't know/no opinion 35 (8 percent of 450)
This example points out several items you need to consider.
First, of the 450 customers asked this question, 22 did not remember receiving the booklet and 35
said they had no opinion or did not know. We presented this information outside the table because
the analyst decided it was more important to focus attention on those who did have opinions. Thus,
the percentages of those with opinions is based on the 393 respondents who expressed them. If it is
important to determine the percentage of customers who don't remember or who have no opinion,
you would calculate those figures using a denominator of 450—the total number who were asked the
question. By including the sample size in the table ( "n = 450"), readers can do these calculations,
should they be interested.
Second, the information presented may be at too great a level of detail for many audience members.
The difference between a "2" and a "3" rating, for example, may not be meaningful for them. Thus,
you want to collapse the information into some smaller number of categories. One possibility is to
create three categories: dissatisfied, neutral, and satisfied. Scores of 1 to 2, 3 to 4, and 5 to 6 might
be collapsed to create three categories and then report:
Customer Satisfaction with the ABC Booklet (n =450)
Rating
dissatisfied
neutral
satisfied
Total
Number
69
254
70
393
Percent of those
expressing an opinion
18
65
18
101*
* Total is greater than 100 due to rounding
don't remember receiving the ABC Booklet: 22 (5 percent of 450)
5-4
-------
don't know/no opinion 35 (8 percent of 450)
Note that the information can now be grasped much more immediately. It is reasonable to ask: If
you will eventually collapse responses, why give customers six possible answers? Research has
shown that people prefer to have a fairly wide range of responses because they don't like to be
"forced" into a Procrustean set of options. In addition, analysts may have different approaches to
collapsing categories.
The responsibility for making information manageable and understandable falls to the analyst. It is
the analyst's task to identify sensible ways to collapse categories and to present these decisions to the
audience (often as a footnote or technical appendix).
Third, as discussed in the next section, you should consider how to present the data. Although these
tables are simple and easy to interpret, compare them to a chart that summarizes the information
instantly.
Customer Satisfaction with the ABC Book
Fourth, the analysis you anticipated during the planning phase should guide whether you need to do
"subgroup analysis," which examines whether different kinds of customers have different kinds of
responses. Suppose you want to examine whether educators and representatives of advocacy
organizations have the same or different opinions about the ABC booklet. You could collapse
categories and sort respondents by their status as educators or advocates (to be sure, some
respondents may be both educators and advocates, but for simplicity, let's assume you had customers
indicate their primary role). You might present the findings this way:
5-5
-------
Selected Customer's Satisfaction with the ABC Booklet
Educators Advocates
Dissatisfied
Satisfied
Neutral
Selected Customers' Satisfaction with the ABC Booklet (n = 450)
Rating
dissatisfied
neutral
satisfied
Total
Educators
Number
27
94
35
156
Percent
17
60
22
99*
Advocates
Number
17
78
12
107
Percent
16
73
11
100
! Total is less than 100 due to rounding
This table provides important information, but you might want to present it using charts for the two
separate groups. You could also perform a statistical test to see if the two groups differ statistically
in their satisfaction with the ABC Booklet.
Fifth, consider the adequacy of your findings. Be sure how strong your findings are before
formulating recommendations. Many factors affect adequacy, such as the sample size, response rate,
and objectivity of questions posed — as well as the way you will use the findings. With a sufficient
sample size, a good response rate (more than 75 percent for mail and telephone surveys, for example),
and questions that are not biased, you can use the information with confidence. OMB requires an
80 percent response rate for survey results to be considered statistically valid. However, when less
than 80 percent of those sampled return questionnaires, the information gathered should still be used
5-6
-------
to improve customer service. Do not ignore the findings.
Let's say that in the above example, an additional 17 small business owners responded to your survey.
This small number may make the sampling error for this group quite high. Nevertheless, pay attention
to the results.
Even if they do not adequately represent the larger group of small business owners who were your
customers, you can still:
- Decide whether the findings are suggestive (rather than definitive). Should your office pay
attention to the concerns suggested by these findings?
- Compare the findings to other similar data. Are small business owners generally pleased or
displeased with other organization products?
- Compare the findings to information your organization gets from continuous feedback
methods. If you call small business owners after providing a service or product, what do they
have to say in those conversations?
- How do the continuous feedback findings compare with the results of this survey?
- Discuss the findings with colleagues. Have they gotten similar reports? Is there a pattern
emerging about small business owners' level of satisfaction with your organization's products?
- Raise the findings with program managers, being careful to note that this might be an area
that requires attention to improve customers' satisfaction with your organization.
- Investigate the findings further. Should you use this as a starting point for more in-depth
discussions with small business owners? Should you conduct focus groups to see how
products could produce higher levels of satisfaction?
One final comment on this example. Your organization may have customer bases much smaller than
the thousands used in the example. If your customer base is quite small, you should decide whether
a statistical sample and quantitative survey is viable, since other techniques may be more suited for
your purposes. If you decide to go ahead with a quantitative survey, recognize that the analyses you
conduct should be carefully considered and constructed. If, for instance, you have 500 customers and
survey 100 of them, you can perform the same analyses as in the example above, but you should
examine the frequency distribution first. In an extreme case, let's assume that 10 of your 100
respondents gave a score of "0," 60 gave a score of "3," and 30 gave a score of "6." Although the
average score of 3.0 may be close to the average of 3.5 in the example, the distribution of responses
is very different.
Sixth, you need to consider how past responses compare with the new responses, and ensure that you
can compare the most current results with those you expect from future questionnaires. This is time
series or trends analysis and is vital to measuring change.
5-7
-------
Driver Analysis
One useful approach in customer research is driver analysis, which identifies the service or services
that most significantly affect respondents' satisfaction. Driver analysis can help you prioritize
findings, which is important because customer feedback efforts often yield more information than an
organization can deal with. Also, managers often don't have enough resources to adequately improve
all areas receiving low satisfaction ratings. Driver analysis enables you to identify which areas
deserve the highest levels of attention.
As an example, let's assume that you are assessing three ways of providing information: by
telephone, by mail, and through published materials. By analyzing customer feedback, you can
identify which method results in the highest satisfaction rates. This is the delivery system that most
strongly "drives" satisfaction with the program's information services. When you identify the method
that significantly affects satisfaction, additional analysis can determine which factor within that
method most significantly affects satisfaction. Continuing with the example, let's assume that you
identify "information received by telephone" as the method producing highest satisfaction. Digging
down another level, you can use driver analysis to identify the factor that most affects the
respondent's opinion. Such factors may include: the accuracy of the information, the courtesy shown
by the employee, or the accessibility of the correct person to answer the question. Identifying the
driver in this way greatly enhances a manager's ability to set priorities for improvements.
You will use two primary analytical techniques, stated importance and derived importance., in driver
analysis:
Stated importance uses respondents' answers to specific questions regarding the importance of the
services. Simply ask the respondent to rank or rate items on a prescribed scale (as from 1 to 6)
according to their importance.
Derived importance uses multi-variate analysis to identify the most important factors affecting
satisfaction. In short, the overall level of satisfaction with the organization is compared to satisfaction
levels of particular products or services received. Driver analysis will identify the degree to which
variation in the overall level of satisfaction is explained by the variation in the product or service
received. Those individual products or services that most adequately explain the variation in overall
satisfaction are the drivers.
Presenting the Data
Before presenting the data, you must remove all identifying information. To ensure credibility and
confidentiality, you should never present findings that could be used to identify a specific customer.
Typically, you would strip names, addresses, and telephone numbers from the analytical database and
keep them in a separate file that includes the unique identification number assigned during data
collection. If ever warranted, you can link the file with identifying information through the
identification numbers.
5-
-------
Most people want the "bottom line," presented as succinctly and clearly as possible. Therefore,
consider presenting your survey results in simple, straightforward ways to most audiences, saving the
mathematical details for an appendix or supplementary briefing. Many audience members want abrief
summary of the study's findings. Two pages of text, with key findings presented as bullets, are
usually sufficient.
Color bar graphs, pie charts and other illustrations can display your findings in a powerful way,
making it easy for your audience to grasp information.
Making Recommendations Based on the Data
Your customers may suggest many potential improvements or enhancements to your permitting
program. You probably should reduce the list to those that will most directly affect overall customer
satisfaction. Most organizations have limited staff and other resources, so practical considerations
must guide their choices. Usually, three to five targeted improvements are sufficient. Sometimes,
a single improvement can have a major impact.
You will have to consider your organization's own capacity for action. However, it is important to
do something. Otherwise, customers may feel that they wasted time providing input that you didn't
value.
Recognize too, that not everyone will be ready for the feedback results. Presenting them can raise
sensitive issues within your organization. Some co-workers may feel threatened by anything but
glowing results. Some may question the credibility of the findings, especially if they build logically
to recommendations for changes that affect them.
To get buy-in and use the results to influence change, results must be honest, and presented in a
constructive way that emphasizes the positives. Results, findings and recommendations should be
presented as opportunities for improvement. If the survey cannot be used to influence change or
improvement, it did not meet its objective, no matter how carefully the feedback activity was
conducted.
Presenting Recommendations - Using Graphics
Remember, at least 70 percent of your message is visual, so take advantage of how people absorb
information. Use the right visuals to communicate your message.
• emphasize main numerical facts
• uncover facts, trends, comparisons and relationships that might be overlooked in text or table
• summarize, group or segment (stratify) data
add variety and interest to text, tables and briefings
5-9
-------
It's best to use pie charts to display components or parts of a whole. Use line charts if you want to
show independent or cumulative values when:
• your data cover a long period of time and several series are compared on one chart
• you want to show change, not quantity
• to exhibit trends
• to show relationships
Do not use column charts for comparing several data sets, for showing data with many plottings, or
to show many components. Finally, use picture graphs to demonstrate concepts or ideas.
ACT ON THE RESULTS
Is this the beginning or the end of the process?
When your efforts to collect customer data appear to be coming to a close, your real work may just
be starting! If this is the first time your organization has collected and analyzed customer data
systematically, you are probably discovering a whole new world of information. Depending on the
feedback method you have chosen, you may have created a baseline of information that characterizes
how your customers evaluate your products and services. You may wish to repeat the same process
again, to measure improvements against the baseline.
Customers will expect you to not only act on their feedback, but also to tell them what you have
done. At the same time, your organization will want to make the best use of the information it paid
to collect. Therefore, this next stage of the process is vitally important to the success of the final
phase-action planning and implementation.
How do you decide what to do with the feedback you receive?
Once you receive and analyze the feedback, most people will be anxious to know the results. How
did we do? What's the bottom line? Work hard to avoid giving answers that over-simplify the
feedback you have received. Depending on the methodology you used, you may have an average
score or rating to report. However, your information probably will provide a wealth of additional
insights about how your customers view the products and services they have received from your
organization.
How good is good enough?
That is a very hard question to answer. In fact, the only real way to answer it is to say "it depends."
For example, is an average score of 4.9 on a 6-point scale a good score? If lastyear's average score
was 2.5, you may have reason to celebrate. For one thing, your score nearly doubled. Even better,
it leaped from the dissatisfied range to the middle of the satisfied range. However, you may want to
look deeper: how do your customers rate others who provide similar services? Is that organization
getting ratings above or below the 4.9? And what about the distribution of ratings—are some
5- 10
-------
customers still rating you below a 3.0 while some are rating you above a 5.5? Are the more positive
ratings obscuring the negative ones? If so, you still may have customers out there who are sharply
critical of your products and services.
Setting acceptable goals for customer satisfaction ratings is a decision that each organization must
make for itself. Keep in mind, however, that leading service organizations:
• Target overall satisfaction scores at the upper end of the scale. On a 6 point scale, that should
be a 5.0, and in very competitive environs it may even be at the 5.5 level or higher.
• View any less-than-satisfied ratings as being unacceptable because they indicate an opportunity
for dissatisfied customers to quickly convey their dissatisfaction by word of mouth. In the long
run, that can undermine your efforts to achieve a reputation for service and product excellence.
How do we know what to work on first?
Many organizations are overwhelmed with the amount of information they receive from customers.
This is especially true if a survey instrument is lengthy, or if the results contain many open-ended
comments and ideas. Decision makers, particularly at more senior executive levels, are likely to ask:
What do we do first? What improvements will yield the best improvement in overall customer
satisfaction? What investments are worth making?
During the planning phase, you, your colleagues, and managers will have identified potential methods
and procedures for acting on the results of customer feedback activities.
Recovery. Be prepared to hear from customers who report a negative experience with your
organization. Set up a quick alert and response mechanism for any such case. (That may require a
special question asking whether the respondent is willing to be identified and contacted for follow-
up.) A quick response is a very positive way to convert a negative impression into a positive one.
Report. Even if the primary means for action is an oral briefing, having written documentation for
others to read and refer to is a good idea. It also creates a historical record for tracking changes over
time. Most people will want to see graphics and summary tables. Reports may include an executive
summary, a description of the study objectives and data collection methods, a comprehensive
investigation of findings (illustrated with graphs and tables), and conclusions and recommendations.
To keep the report at a reasonable length, you can present supplementary material in appendices.
Brief. Action planning workshops get management's attention. Gather decision makers together and
go over the findings with a verbal presentation. Software graphics packages can help make the
briefing interesting and informative. Conducting a dry run before your presentation helps with timing,
pacing, and finding out how well you can verbally communicate your written findings. Hard-copy
handouts give participants a tangible reminder of the information conveyed.
5- 11
-------
Prioritize. Try to package the information so that it leads the audience or reader to a series of
practical actions that fit logically. Acting on results may be more successful if several smaller action
plans are developed that contain three to five steps, rather than one large plan that may appear
overwhelming.
Communicate. In addition to briefing management, it is a good idea to communicate results to
others. Sending a thank-you letter to focus group participants and customers should note what your
organization learned and what will be done with the findings. Your employees are often eager to
learn what customers have said, so results should be summarized and distributed widely.
Improve. There is no reason to elicit customer feedback unless you will use the information to
improve your organization's processes, services, or products. Recognize that some employees may
be excited about possible changes, yet others threatened and resistant. The best way to use customer
feedback may be to develop and define action plans. They are most likely to be successful when
"owners" of each issue:
are identified and included
• help assess their activities and customers' feedback
• participate in review and strategy sessions
• have an opportunity to discuss concerns and shortcomings in nonthreatening, non-confrontational
environs.
Enhance. Sometimes customers are satisfied, but want the agency to do more. This must be seen
as an opportunity to enhance products or services.
Reward Conducting customer feedback activities can be exciting and worthwhile; the process can
also be exhausting and threatening. Be certain that you recognize and reward the efforts of staff and
customers who made the activity possible. Rewards can take the form of public acknowledgment,
mention in performance reviews, and attention to findings.
Plan. Use customer feedback to see what worked well and what could be improved the next time
around. Identify aspects that facilitated or impeded achieving the project's objectives, including
aspects of planning, data collection, analysis, and development of findings.
Feed Results into the Strategic Plan and GPRA Goals and Planning Activities. Recent
management initiatives, including the President's directives on strategic planning, reinvention, and
customer service improvement, and the Government Performance and Results Act (GPRA), suggest
that customer data be included in performance data. To address these needs, quantitative data from
surveys and trend data accumulated from on-going feedback mechanisms may be most useful. You
5- 12
-------
can use focus groups and other qualitative data to clarify customers' views.
As government agencies go about "reinventing" programs to meet customers' needs and expectations
and to comply with the requirements of GPRA, managers will need to develop customer-based
performance goals and indicators to assess progress. The basic way to do this is to get input directly
from customers.
5- 13
-------
Chapter 6
Maintaining Good
Customer Service
This chapter of the Toolkit discusses some of the ways an organization can
keep its customer service program effective and viable.
Maintaining an effective and viable customer service program requires you to pay attention to several
factors. You can=t achieve good customer service by implementing a one-time initiative. It requires
you to institutionalize the basic tenets and values of customer service throughout your organization
— and to regularly reinforce them. Some of the key elements that we recommend:
Establish a customer-driven organizational culture. Make sure
that the benefits of customer service are understood and appreciated
throughout your organization, that all staff feel empowered to practice
and improve customer service, and that customer service
accountability, recognition, and award systems are in place. Your
organization should have a customer-oriented mission, vision, and
organizational values. Your organizations commitment needs to be
visible and obvious to your staff. You can accomplish this through:
posters, placing customer service on meeting agenda, regular LAN or
e-mail messages, etc. You may need to assess staff attitudes and
assumptions about customer service within your organization and
appropriate corrective steps.
Maintain management support and commitment. Management
must commit the necessary resources to customer service. The
organizations strategic planning documents, such as a permit
programs annual operating plan, should reflect a commitment to
customer service. Your organizations policies and practices should
be flexible, adoptive and resilient.
Know the processes and tools of customer service. Managers and
staff need the skills to implement customer service programs and
maintain/improve customer satisfaction. Below is a service skills
matrix developed by and included here with the permission of, Macro
International, Inc. of Calverton, Md., to define the desired skills.
Training programs can help your staff understand why customer
6- 1
-------
service is important and how to achieve customer satisfaction in your
organization. A maj or telecommunications company has reported that
customer satisfaction rose from 75 percent to 93 percent after its staff
received customer service training. EPA has developed a training
program that includes a 2-1/2 hour introductory workshop and six
additional modules tailored to specific situations. EPA staff are
conducting these workshops around the country. The introductory
workshop, entitled "Forging the Links," encourages participants to
unleash their creativity to enhance EPA customer service. The six
additional modules are entitled: "The Leader in Each of Us;" "Moving
From Conflict to Collaboration;" "Proactive Listening;" "Clarifying
Customer Expectations;" "Resolving Customer Dissatisfaction;" and
"Influencing for Win-Win Outcomes." These modules help EPA staff
develop and use the personal skills they need to provide outstanding
customer service. You can obtain more information on this training
by contacting Patricia Bonner at EPA Headquarters, telephone
number (202) 260-0599.
Establish accountability for customer satisfaction. Managers and
staff need to understand and be held accountable for implementing
customer service programs. The most direct way to accomplish this
is through performance standards and performance appraisals.
Keep in touch with your customers. Regularly request, evaluate,
and respond to your customers: needs. This means obtaining feedback
through any of the approaches described earlier in this toolkit.
Respond as appropriate and keep them informed of improvements
designed to meet their needs. A Web page or newsletter are excellent
vehicles to reach your customers with such information.
Institute appropriate incentives, particularly recognition and/or
award programs. Staff excellence in customer service needs to be
promoted, recognized, and reinforced. Incentives can range from
letters of appreciation to cash award programs. We recommend the
book, A1001 Ways to Reward Employees," by Bob Nelson, as an
information source for creative incentive programs.
6-2
-------
Service Skills Matrix
u) o
01 U)
in zr
CD £
HANDLING CUSTOMER INTERACTIONS
Ability to define customers
Knowledge of what customers expect
Skills in providing responsive service to customers
Knowledge of how behaviors trigger productive and unproductive
customer interactions
Skills in reducing stress in tense customer interactions
Skills for resolving difficulties with angry customers
Customer problem-solving skills
Ability to respond to verbal attacks
Skills in saying 'no' to customers in a productive way
X
X
X
X
X
X
X
X
X
U)
II 1
LJJ
^r
_i
Q.
;>
LJJ
X
X
X
X
X
X
X
X
X
6-3
-------
Customer Service
Skills Matrix
(Continued)
GERS&
{VISORS
SKILLS, KNOWLEDGE & ABILITIES ^^^H
COACHING & MOTIVATING CUSTOMER SERVICE
BEHAVIORS
Ability to create and communicate specific, measurable, customer-
focused performance
Ability to establish customer service performance standards for
employees
Skills in delivering effective, constructive feedback to employees
about their customer service skills
Ability to identify and evaluate performance problems, and to
determine how and when to take corrective action
Ability to reinforce desired staff behaviors with customers
MANAGING CUSTOMER SERVICE IMPROVEMENT
Knowledge of what drives customer "loyalty" in the context of
public organizations
Skills in analyzing customer transactions (cycles of service)
Ability to gather, analyze, and respond to customer feedback
Ability to use customer feedback to improve service processes
Ability to establish "service recoveries"
Ability to determine appropriate levels of service delivery
empowerment for subordinate employees
X
X
X
X
X
X
X
X
X
X
X
w
LU
LU
Q.
5
LLI
6-4
-------
Customer Service
Skills Matrix
(Continued)
SKILLS, KNOWLEDGE & ABILITIES
USING CUSTOMER FEEDBACK
a: v)
in ;r
%*
< UJ
^ Q-
w
LJJ
in
O
LU
Skills in using focus groups in the context of customer service
improvement efforts
X
GATHERING CUSTOMER FEEDBACK
Skills in developing surveys, interviews, and other methods for
obtaining specific, actionable, and timely data
X
Ability to administer a survey and interview processes
X
Ability to analyze and report customer service data
X
Ability to interview customers to obtain feedback
X
6-5
-------
Customer Service in t Jj* Chapter 7
^Sk-7. Perm ittinsssifl Customer Service
in Action
TTz/s chapter of the Toolkit offers some examples of what
environmental agencies are doing to improve customer satisfaction.
Introduction
This Toolkit presents an approach to developing world class customer service in permitting programs.
While we have described a process, and hopefully given you the tools to identify and evaluate your
customers: needs, we cannot prescribe the right solutions for your organization. Nevertheless, we
believe it is useful to offer some real-world examples of what organizations like yours are doing. This
chapter may inspire you as to what=s possible, and give you some leads on whom you may contact
for further information.
This is by no means an exhaustive account of what is happening across the country and it may not
be completely up-to-date, but we think it reflects some of the more interesting and innovative efforts
we=ve learned about during our work.
Permit Assistance/Information Center
States offer a variety of assistance to permit applicants and citizens who want help with permit
information and application processing. The most common approach was establishing either a permit
information center or a designated coordinator to serve as a single point of access to the regulatory
agency.
Permit assistance offered by one or more states:
$ Answering general phone inquiries and referring, as needed, to the appropriate program
permitting staff [Maryland, Missouri, North Carolina]
$ Helping the permit applicant identify the types of permits, certifications, or approvals it needs
[North Carolina, Oklahoma, Vermont]
$ Coordinating pre-application meetings between the permit applicant and the program offices
[Maryland, Missouri]
$ Helping the permit applicant evaluate alternative permitting processes [Vermont]
7- 1
-------
$ Serving as a repository for all permit information; distributing guidance materials, permit
assistance directory, and application forms for easy distribution to the public [North Carolina]
$ Establishing a tracking system that provides the public/applicant with fast information on the
status of permit applications [Maryland]
Stakeholders' Early Participation in the Permit Process
Citizens often believe agencies make major decisions on a permit long before they seek public input,
or that the public doesn't have enough time to properly evaluate a permit application.
They may raise issues at the draft permit stage that could have been more readily, and less
expensively, resolved earlier in the process. To respond to these concerns, some organizations are
soliciting earlier stakeholder involvement than typically occurs.
For example, EPA Region 2 implemented a Permit Complaint System Policy with enhanced
stakeholder communication procedures. The policy includes enhanced public access to permit
information, including Internet access, mailings to stakeholders on a permit-specific basis, early
identification of issues, and expedited complaint resolution earlier in the permit process than
prescribed under regulation.
Among states, Connecticut and Maine established an early public involvement program to seek input
from the permit applicant and public at the beginning of the permit process. Under Connections
Early Public Participation Program, a public notice informs citizens that the agency has received a
completed permit application. Citizens may request a public meeting to discuss the proposed
application before any decision is made by the agency. In Maine=s case, a team of project managers
meets with potential applicants at the early stages of the permitting process. This pre-application
meeting allows both parties to identify any applicable regulations, and even analyze and resolve
potential issues before the official application is filed with the regulatory agency.
While Connecticut and Maine seek all stakeholders: input on the permit application process, South
Dakota seeks input from the public and the regulated community on proposed new regulations. The
goal is to seek early input from all stakeholders for applicability, practicality and enforceability before
rules are promulgated.
Plain Language
The federal government is working to improve the clarity of regulatory language in its letters and
communications by using plain English. In a similar effort, South Dakota has a "Plain English"
program underway. Under this program, South Dakota=s written and oral communications are made
easier to understand than the usual bureaucratic or technical jargon. Through this program, even
enforcement letters are written in simple language so that violators better understand what needs to
be done. South Dakota believes using plain English helps small business owners better understand
7-2
-------
what the agency is doing, what it requires, and what services it offers. It may even boost compliance.
Permit Processing/Issuance Timeframes
Currently, most permit applicants don=t know when an agency will make a decision on their permit.
Such unpredictability may create the perception that the state is not responsive to the needs of the
business, which, in turn, creates frustration on both sides.
Consistent with one of EPA=s aforementioned customer service standards for permitting, several
states have passed laws requiring their agencies to establish time frames for processing permit
applications and issuing certain final permits. Maine, Maryland, Hawaii, and Vermont have set either
a "guaranteed" or a targeted "goal" timeline on their permit decision processes. The "goal" takes into
account a number of factors, including complexity of the permitting process, extent of public
involvement, permit backlogs, or lack of program resources.
For most major permits, Maryland, requires agencies to establish money-back guarantees of permit
application fees if they fail to issue permit decisions by the dates promised. In Hawaii, the permit is
automatically approved if the agency fails to act within the prescribed time.
Other states, including Oklahoma and Connecticut, have set permitting timeframes without
legislation. They have established permitting timeline goals for regulated communities. The states
say this approach has reduced permit processing timeframes and, consequently, improved customer
satisfaction.
To measure the success of its agency in meeting the established timelines, Vermont's legislature
required the Vermont Department of Environmental Conservation to generate annual reports on how
they are doing and ways to correct problems.
Maryland requires its program offices to generate success reports on adherence to permit processing
timeframes.
Customer Satisfaction Surveys
Several states are conducting surveys to assess their customers: satisfaction. Maine, Oklahoma, and
Delaware use mail surveys to assess satisfaction with their staff s courtesy, timeliness, knowledge and
quality of service. Missouri, North Carolina and Vermont surveys measure satisfaction with the
permitting process. These agencies include a customer survey with each individual permit.
While the above-mentioned states primarily survey their external customers, Wisconsin also conducts
internal surveys of its permitting staff for continuous quality improvement of its processes. They
gather information on how and what data are used for permitting decisions. Wisconsin believes that,
by understanding staff needs, they will enable the state to focus on specific areasissues, which should
improve service to their external customers.
While most of the states conducted surveys annually, Vermont experienced some poor response rates
7-3
-------
and has decided to revamp its processes. One option may be conducting a more intensive effort, but
less frequently.
Incentive/Award/Reward Programs
EPA Region 6 has implemented an innovative and effective employee recognition system for
excellence in customer service. It includes initiatives such as Peer Recognition Certificates with time
off awards, a Monthly Customer Service Awards program, the Regional Administrator's Annual
Customer Service Award, the Division Director Customer Service Awards, and a Mystery Caller
awards program to improve voice mail messages.
Delaware implemented monthly Customer Service Awards to recognize staff for providing
outstanding service to internal and external customers. Monthly award recipients are selected by
customer service representatives from each of the divisions sections.
7-4
-------
APPENDICES
A: Internal Control Procedures
B: Sampling
The Basics
• More on Sample Size
C: OMB Clearance
D: Model Surveys
• Applicant Permitting
• Citizens Permitting
• Delegated Authority Permitting
-------
Appendix A: EPA Internal Control Procedures
Internal Controls Ensure the Integrity of Survey Data and Results
The U.S. General Accounting Office and OMB have issued Internal Control Standards that apply to
all operations and administrative functions, thus ensuring the quality, reliability and integrity of
information we use for decision making. These standards and techniques, found in a number of laws
and requirements, apply to the collection, administration and reporting of customer survey results
and other forms of data used for performance measurement, verification, planning and management
action. Developing internal control procedures is good business practice and important in our role
as public servants. What constitutes an effective control system varies with your circumstances.
While controls may be as routine as second party reviews or limiting access to the data, they should
provide reasonable assurance that you will reliably and cost-effectively accomplish the survey's
obj ectives. Any audits, evaluations or verifications of your data will usually start with an examination
of internal control systems.
Some Specific Control Standards and Techniques
Management must provide reasonable assurance and a supportive attitude to protect assets
(information) against waste, loss, unauthorized use, and misapplication. Supporting documentation
should be clear and available for examination. Management controls should be logical, applicable,
reasonably complete, efficient and effective in accomplishing management objectives. Managers and
employees must have professional and personal integrity and are obligated to support the ethics
program and maintain a level of competence that allows them to accomplish their assigned duties.
Managers should ensure that appropriate authority, responsibility, and accountability are defined and
delegated and that an appropriate organizational structure is established to effectively carry out
program responsibilities. Managers should spread key duties and responsibilities in authorizing,
processing, recording, and reviewing official information to avoid concentration of power and to
discourage abuse of authority. Access to assets and records should be secured and limited to
authorized personnel, with custody assigned and maintained. All program operations, obligations and
costs should be in compliance with applicable laws and regulations; resources should be used
efficiently and duly authorized.
A- 1
-------
Appendix B(i): Sampling - The Basics
If you have decided to use a survey approach for obtaining customer feedback, you need to determine
what sample size to use. This Appendix first discusses sample sizes, sampling error, and confidence
intervals — all of which factor into decisions about the sample size. It then presents a table for
determining sample sizes — and tells you step by step how to use the table. You will also learn how
to randomly select customers for the survey.
What Kinds of Sample Sizes Are We Talking About?
Before we give specific guidelines on how to choose the sample size, we should set some general
expectations. National public opinion polls like the Gallup Poll and the Roper Poll typically use
sample sizes of 1,350 to 1,800. These polls use fairly large sample sizes to obtain a result that
represents the entire adult U.S. population, with a sampling error of plus or minus 2.5 to 3 percent.
Such small sampling errors are needed because the polls often address matters of national importance.
The decisions made, based in part on the results of these national polls, may be far-reaching, long-
lasting, and affect millions of people.
Your surveys to obtain customer feedback will be very different. The target group whose opinions
you need will be much smaller: the people who have come to one specific program within a limited
time (e.g., during one year) to request certain products or services. Your target group may be as
large as 500 to 1,000 people (few EPA programs directly serve more customers than that) or as small
as 50 people or fewer. Furthermore, although the decisions based on customer feedback are
important, they will probably not be far-reaching and long-lasting. In most cases, you will be asking
these types of questions:
• Should we change a process to reflect customer comments?
• Should we revise some of our written products?
• Should we provide a half day of customer feedback training to each staff member?
Even if we make the wrong decision based on a customer survey, we should discover our error soon
enough and be able to correct it without irreparable damage.
Based on these considerations, we can tolerate higher sampling errors than those associated with
national surveys like the Gallup poll. We can feel comfortable with sampling errors of 5 percent or
even 10 percent.
Additionally, you have relatively small target groups who were served by a specific program during
a defined time period. For this reason, you can use a much smaller sample size than is used in the
Gallup and Roper polls, which seek to accurately capture the opinions of millions of people.
B(i) - 1
-------
Sampling Error
"Sampling error" is normally presented as a percentage with a plus or minus sign in front of it. For
example, the sampling error in one particular situation may be ± 3.5 percent. That means that the true
value of a given measure for the entire population—that is, the whole target group you are getting
feedback from—is the value obtained from your sample of customers, plus or minus 3.5 percent. If
for example, 62.4 percent of your sampled customers are satisfied, the actual percentage of satisfied
customers lies between 58.9 percent (62.4 percent - 3.5 percent) and 65.9 percent (62.4 percent +
3.5 percent).
But that is not quite true. In fact, there is no reasonable sample size through which we can be certain
that we will obtain the true value for all customers.
Why is that so?
The characteristics of the customers in the sample may occasionally be very different from the
characteristics of the customers not in the sample. In such circumstances, the true value for all
customers will be very different from the value obtained from the customers surveyed. The only way
to get around this statistical fact is to specify "how certain we want to be" that the true value does,
in fact, fall with a specific range. This degree of certainty is known as the "confidence level."
Confidence Level
The "confidence level" indicates how confident we want to be that the true value lies within a specific
range.
No one confidence level is the "right" one to use. There are many different possible confidence
levels, and only you can decide which confidence level is appropriate for your survey.
Many public opinion surveys use the 95 percent confidence level. That means that you can be 95
percent certain that the true value for all your customers will lie within a specific percentage band
(one equal to the size of the sampling error) around the result you obtain from your sample.
Another confidence level commonly used is the 90 percent confidence level. With a 90 percent
confidence interval, you can be confident that 9 times out of 10, the true value falls within the value
obtained from your sample of customers, plus or minus the sampling error. Some analysts use 80
percent confidence intervals.
To decide what confidence level to use, you might want to think of a scale running from 80 to 95,
where 95 represents a high level of confidence and 80 represents a lower level of confidence. Decide
which confidence level to use based on the way you will use your results, how products and services
may be affected by the results, and the frequency with which you will collect additional information
to confirm or revise your findings.
-------
Determining the Sample Size
Now that we have established appropriate expectations with regard to sampling error and sample size,
we will provide you with some guidance on selecting your sample size. Please recognize that there
are several factors to consider in determining the sample size. The information provided here is
intended to help get you started. Please refer as well to the additional information provided in
Appendix B(i) and Appendix B(ii). If you wish, you may also consult a statistician at EPA. A list
of EPA statisticians showing the EPA Office in which each of these statisticians is located can be
obtained from the Office of the Chief Statistician of EPA within EPA's Center for Environmental
Information and Statistics by calling 202-260-5244.
Number in Target Group
1000
1000
1000
500
500
500
200
200
200
100
100
100
50
50
50
1000
1000
1000
500
500
500
Sampling Error
±5
±5
±5
±5
±5
±5
±5
±5
±5
±5
±5
±5
±5
±5
±5
±10
±10
±10
±10
±10
±10
Confidence Level
80
90
95
80
90
95
80
90
95
80
90
95
80
90
95
80
90
95
80
90
95
Sample Size
141
214
278
124
176
218
90
116
132
62
74
80
39
43
45
39
64
88
38
60
81
B(i) - 3
-------
200
200
200
100
100
100
50
50
50
±10
±10
±10
±10
±10
±10
±10
±10
±10
80
90
95
80
90
95
80
90
95
34
51
66
29
41
50
23
29
34
The above table is appropriate for simple random sampling (SRS), which is a sampling procedure
based on sampling without replacement. Simple random sampling is the most commonly used
sampling procedure. The table is based on the approximate formula given in Appendix B(ii). This
approximate formula includes an adjustment comparable to the finite population correction factor for
each combination of target population and sample size.
The precise formula that can be used instead of this approximate formula is also given in Appendix
B(ii). For a discussion of the finite population correction factor, see Appendix B(ii).
The procedure described below for randomly selecting a sample from the full list of customers served
in a specific period of time is simple random sampling and is therefore consistent with the above
table.
Here's How to Use the Above Table
The instructions that follow assume that your survey's unit of analysis will be the "person served."
(1) Identify the number of persons you have served in a given time period. Find that number in the
column labeled "Number in Target Group."
(2) Select the confidence level that you consider to be the most appropriate given the magnitude of
the decisions that will be made:
> If the decisions to be made using the survey results will be far-reaching, long-lasting and/or
costly, use the 95 percent confident level.
*• If the decisions to be made using the survey results will be less far-reaching, less long- lasting
or less costly, use the 90 percent confidence level.
-------
> If the decisions to be made using the survey results will have more limited consequences,
mostly in the short-term (e.g., in the next 6-12 months) and the cost implications of the
decisions will be moderate, you may use the 80 percent confidence level.
(3) Select an appropriate sampling error:
> For most EPA customer satisfaction surveys, a sampling error of ± 10% should be acceptable.
> In cases where the decision s will require greater certainty and precision, a sampling error of
±5% can be used instead.
(4) Read off the corresponding sample size.
> If the total number of customers falls between two of the values shown above in the column
"Number in Target Group," you can use interpolation to obtain an initial estimate of the
appropriate sample size.
*• You can then use the approximate formula for determining sample size presented in Appendix
B(ii) to obtain a much better estimate of sample size needed.
> You can stop here and use the approximate sample size obtained in step (4)(b) immediately
above. Or, if you wish, you canuse the trial and error approach described in Appendix B(ii),
or even better, the combined approach, also presented in Appendix B(ii), to calculate the
precise value for the sample size needed.
Here's How to Randomly Select a Sample of Customers Once You Have Determined
What Sample Size to Use
Once you have determined the appropriate sample size, you need to randomly select that number of
customers from the total number served in the time period of interest. Here is a procedure you can
use to make that random selection:
(1) Make a complete list of all the persons served in the period of interest for which you already have
(or can obtain, with a reasonable expenditure of effort) the needed contact information (i.e.,
name, plus address or phone number). Put the customers in alphabetical order. Eliminate
duplicates (so that each name appears only once).
(2) Starting at the top of the list, number each name. The result is the "master lisf of customers
served. The number next to each name is that person's "customer number."
(3) Here is a computer-based approach for selecting a sample of customers from the master list:
B(i) - 5
-------
(a) You will use spreadsheet software (such as Lotus 1-2-3 or Excel) to carry out the remaining
steps of this procedure. Before you begin using any particular spreadsheet software, make
sure it has a "randomize" function. Not all spreadsheets do.
(b) Enter the customer numbers in the second column of the spreadsheet, placing the numbers
in numerical order, one number per row. Leave the first column blank. You should have a
spreadsheet with the number of rows equal to the number of customers. In the second
column you should have the numbers "1", "2", "3", and so on (up to the total number of
customers served).
(c) Use the "randomize" function on the second column of the spreadsheet. The numbers in the
second column are now in random order.
(d) Enter numbers into the first column of each row. Enter the number " 1" into this column in
the first row, enter "2" in the second row, and so on. These new numbers are the row labels.
(e) Mark off the number of rows corresponding to the sample size chosen above. For example,
if the sample size is 65, mark off the first 65 rows.
(f) The numbers appearing in the second column are the customer numbers for your sample.
Using the master list prepared in step (2) above, read off the names next to these customer
numbers. Place these names in a new list. These are the people you will contact to answer
the survey questions.
(g) Later, if you have a low response rate to your survey despite reasonable follow-up efforts,
go back to the spreadsheet and mark off the additional number of rows needed to reach the
desired sample size. The numbers appearing in the second column of these additional rows
are the customer numbers for the additional customers to be added to the sample.
B(i) - 6
-------
Appendix B(ii): Sampling - More on Sample Size
The Effect of the Response Rate on Sample Size
The initial sample size is the number of customers you attempt to contact during the survey. The
final sample size is the actual number of customers for which responses were received. The response
rate is the percentage of customers included in the initial sample for which a usable response was
received. The response rate will vary depending on the kinds of customers being contacted, the kind
of product or service received, the kinds of questions asked in the survey, and so on.
The response rate is almost always less than 100 percent. The table in Appendix B(i) shows the
approximate sampling error associated with the final sample size. You should always include a
greater number of customers in the initial sample to achieve the desired final sample size.
For periodic surveys that ask the same or similar questions (in order to measure changes in customer
satisfaction) you can estimate response rates based on rates in previous surveys. If you are conducting
your survey for the first time, you could assume a response rate of, say, 85 percent when determining
how many customers to select for the initial sample. If fewer than 85 percent respond, you can add
more customers to the sample later, using the procedure described in step (3) (g) in on pages 5-6 of
Appendix B(i).
Note, however, that it's better to have high response rates and among a smaller number of customers
in the sample. The reason for this is non-response bias. Non-response bias is encountered if the
customers who did not respond to the survey are significantly different from those who did respond.
Non-response may be due to your inability to reach a specific customer, (e.g., because his or her
telephone number has changed), or may be due to the customer's unwillingness to participate in the
survey at all, or to answer one or more questions in the survey. Because some customers will answer
some questions but not others, the degree of non-response will vary from question to question on the
survey questionnaire.
Non-response bias is one source of the overall bias in the survey results. Another source of bias is
using a poorly chosen or poorly constructed master list from which we randomly select the sample
of people to be surveyed. One of the best known examples of such bias is a classic national poll of
likely voters that was conducted by the Literary Digest in 1936, a few days before the presidential
election that year. The poll showed that Alf Landon would win the election. In fact, Franklin
Roosevelt won the election by a landslide. The reason for the erroneous polling results was bias. The
poll was conducted relying primarily on lists of telephone subscribers. Since 1936 was the lowest
point of the Great Depression, many voters could not afford phone service. It turned out that voters
without phones were much more likely to vote for Franklin Roosevelt than those who had them.
While this particular case gives an unusually dramatic example of bias, any level of non-response (like
any serious systematic errors in preparing the master list) poses potentially serious problems.
Furthermore, we won't know the magnitude of these problems because we probably won't know how
the non-respondents differ from those who did respond. After all, we were never able to gather any
information about them in our survey.
B(ii) - 1
-------
For this reason, you should keep non-response rates to the lowest achievable levels. You can do this
by following up with customers who didn't respond to your first attempt. Only after you've made
all reasonable follow-up efforts should you make up the shortfall by selecting additional customers.
An Adjustment Factor
The sampling errors shown in the table presented in Appendix B(i) are approximate. They do not
take into account a factor that, if considered, would result in lower values. We will now provide you
with an adjustment factor that you may use to account for this additional factor and, in so doing,
obtain a more precise value for the sampling error:
An adjustment factor to reflect that the sample result was greater than or less than 50 percent
One significant complication is that the sampling error varies markedly with the magnitude of the
sampling result obtained. By sampling result, we mean, for example, the percentage of customers
in the sample who say they are satisfied with the product or service they received. All else being
equal, the largest sampling error is associated with a degree of satisfaction of exactly 50 percent.
Any higher or lower level of satisfaction will result in a lower sampling error. The lowest sampling
error is associated with a level of satisfaction of 100 percent or 0 percent.
Here are the specific correction factors you should use for various sample results:
The Sample Result (e.g., the percentage
of customers in the sample who said they
were satisfied with the product or
serviced received) Correction factor
99% 0.20
98% 0.28
95% 0.44
90% 0.60
80% 0.80
70% 0.92
60% 0.98
50% 1.00 (i.e., no correction)
40% 0.98
30% 0.92
20% 0.80
10% 0.60
5% 0.44
2% 0.28
1% 0.20
-------
Thus, if the survey shows that 90 percent of the customers sampled were satisfied with the product
or service they received, then the associated sampling error is obtained by multiplying 0.60 times the
sampling error shown in the standard tables (including the table provided in Appendix B(i). So if the
sampling error shown in the table is ±10% for the sample size used, then the actual sampling error
is really only ±6% (= ±10% x 0.60).
If the sample result shows that 80% of the customers were satisfied, and the sampling error obtained
from the table was ± 10%, the actual sampling error associated with that specific sampling error would
be ±8% (= ±10% x 0.80). These are rather significant adjustments.
Since the levels of satisfaction likely to be obtained for most EPA products and services are likely to
be in the range of 80 to 90 percent or more [questionable assumption?], we advise you to take this
adjustment factor into consideration: (1) when calculating sample sizes , and (2) when determining
the actual sampling error associated with a given survey result.
There is a major implication of this variation in sampling error. Since the sample result varies from
question to question, there is no one level of sampling error associated with the survey as a whole.
Instead, you will have a different sampling error for each for the response to each question. If
customer satisfaction is close to 50 percent on one question and close to 100 percent on another, the
sampling error for the second will be much lower than (possibly less than half of) the sampling error
for the first. The plus or minus figure given should therefore be different for each result reported (i.e.,
it should be different for each question for which the response is shown). It is common practice,
however, for only one level of sampling error to be shown: this may either be (1) the largest sampling
error associated with any of the results reported or (2) the sampling error that would be obtained in
the worst possible case, i.e., if the result had been a level of satisfaction of 50%.
In presenting the results for customer satisfaction surveys conducted at EPA, those preparing the
results may either conform to this common practice or present question-specific sampling errors, as
they prefer. The latter can be accomplished by simply stating a plus or minus figure after each sample
result shown.
For example:
The Question on the
Survey for Which the Result
Is Being Reported
Question 1
Question 2
Question 3
Question 4
Question 5
The Degree of
Satisfaction Reported
83% ± 8%
91% ±6%
78% ± 9%
87% ± 8%
94% ± 5%
B(ii) - 3
-------
Precise Formula for Calculating the Sampling Error
Here is an alternative approach for (1) estimating the sampling error that will occur in a planned
survey or (2) calculating the actual sampling error associated with a specific result in a completed
survey. Instead of obtaining values of the sampling error from a table (like that included in Appendix
B(i)) and then applying the necessary adjustment factor(s) presented above in the previous section,
simply calculate the sampling error directly from the precise formula.
Here is the precise formula for calculating the sampling error:
p x q N -n
The sampling error = (Z) times the square root of x
n N-l
where p = the sample result (i.e., the percentage of customers who
were satisfied with the product or service they received)
q= 1-p
n = the sample size
N = the total number of customers served
Z = is a constant coefficient (i.e., multiplier) associated with
the confidence level used. (This must be looked up in a
table in a statistics book). Each of these constants is
known as the Z-score for that confidence level.
Here are the coefficients (i.e., Z-scores) for the three
confidence levels:
For the 95% confidence level, Z = 1.90
For the 90% confidence level, Z = 1.645
For the 80% confidence level, Z = 1.282
The precise formula presented above is based on the simple random sampling (SRS) procedure, in
which the sample is drawn using the sampling without replacement procedure. Simple random
sampling is the most commonly used sampling procedure for use in customer satisfaction surveys
conducted by EPA. It is the procedure reflected in the table and the discussion of sample selection
presented in Appendix B(i).
B(ii) - 4
-------
The above formula will give the exact size of the sampling error for any combination of number of
customers served, sample size, sample result and confidence level. Using this formula automatically
takes into account the differences in the magnitude of the sampling error due to differences in the
sample result (which was discussed in the previous section of this Appendix) and also automatically
includes the finite population correction factor, which is discussed in the next section of this
Appendix.
Another Adjustment Factor
The table presented in Appendix B(i) reflects both the sample size (n) and the total number of
customers served (N) in determining the sampling error for any given confidence level you select.
If you consult reference books on statistics or sampling procedures, you may find tables that show
sampling errors for various sample sizes without factoring in the total number of customers served.
In such cases, to get the actual sampling error, you must multiply the sampling error given by an
additional factor known as the finite population correction factor.
The standard sampling techniques were developed for sampling a very large number of people. This
is true, for example, of surveys of national public opinion. The standard formulas and tables used are
therefore predicated on sampling from a very large pool, one that is, in practical terms, "as good as
infinite" and is treated by statisticians as though it were infinite.
When the number of people in the target group is much smaller, you should use a correction factor
(one known as the finite population correction factor) to correct for this circumstance. The finite
population correction factor can always be used (its use never gives an incorrect result), but you
generally don't need it if your sample size is less than about 10 percent of your target group.
If the sample size of customers to be contacted is greater than 10 percent of the total number of
customers served, then you should use the finite population correction factor in calculating the
sampling error. These circumstances will apply in a large percentage of customer satisfaction surveys
conducted by EPA. Luckily, the finite population correction factor always results in a lower sampling
error. Therefore, if you are satisfied with the sampling error calculated without using the finite
population correction factor, then there is no needto use it for that survey, unless you want to know
exactly how much lower the true sampling error is.
The finite population correction factor (FPCF) can be calculated using the following formula:
FPCF = the square root of (N-n)
(N-l)
where: N = the total number of customers served
n = the number of customers in the sample (i.e., the sample size)
B(ii) - 5
-------
The corrected sampling error is obtained by multiplying the finite population correction factor by the
sampling error obtained from a standard table that considered only sample size and confidence level
(and did not consider the size of the target group (i.e., the total population)). The adjustment factor
becomes more significant as you increase the percentage of the target population you sample. See the
following table:
Sample Size as a Fraction (Percentage)
of the Size of the Target Population Approximate Value of the
= n/N Finite Population Correction Factor
10% 0.95
20% 0.89
40% 0.77
50% 0.71
60% 0.63
70% 0.55
75% 0.50
As can be seen from the above table, if the sample size is approximately 10 percent of the target
group, then the correction factor is approximately 0.95. Thus, when you sample 10 percent of your
customers, the sampling error will be reduced to 95 percent of what it otherwise would have been
(i.e., the sampling error would be reduced from ±10 percent to ±9.5 percent).
If you sample 20 percent of the target group, then the correction factor is approximately 0.89 —
reducing the sampling error to 89 percent of what it otherwise would have been (e.g., the sampling
error would be reduced from ±10 percent to ±8.9 percent).
If you sample 50 percent of the target group, then the adjustment factor will be approximately 0.71
— reducing the sampling error to 71 percent of what it otherwise would have been (e.g., the sampling
error would be reduced from ±10 percent to ±7.1 percent).
There is a general rule of thumb used by many statisticians: the finite population correction factor
should be applied whenever the sample size is 10 percent or more of the target group from which the
sample is to be drawn.
Note, however, that using the finite population correction factor will always give you a more accurate
value for the sampling error. Therefore, you should never be reluctant to use it. Under certain
circumstances (i.e., when the sample size is less than 10 percent of the target population), you can
disregard it (i.e., not apply it) without placing undue adverse effect on the estimated size of the
sampling error.
Note also that the last element in the precise formula for calculating sampling error given in the
previous section of this Appendix is the finite population correction factor. By using that precise
B(ii) - 6
-------
formula, you can ensure that the finite population correction factor is automatically taken into account
when determining the size of the sampling error.
One final technical note: you may have noticed the second column in the table above is labeled the
approximate value of the finite population correction factor. The factors are not exact because we
used the following approximation to calculate the values shown in the second column:
Instead of using the precise formula for the finite population correction factor:
FPCF = the square root of N - n
N- 1
The following approximate formula was used:
FPCF = (approx) = square root of N - n
N
For most values of N (the size of the target population), the difference between the true value
obtained from the precise formula and the approximate value obtained from the approximate formula
is very small.
A Trial and Error Procedure and An Approximate Formula for Determining Sample
Size
You can use the precise formula given above to determine the sampling error for any combination
of confidence level, number of customers served, and sample size. You can use that same formula
to determine sample size when you know the desired confidence level, the desired maximum level of
sampling error and the number of customers served. Unfortunately, you can't use it directly to obtain
sample size in such situations. This is because sample size (n) appears two different places in the
equation, and the equation can't be rearranged to solve directly for sample size. Instead, you must
use the precise formula indirectly to to determine the needed sample size. You can do so as follows:
(1) Begin by guessing what the needed value of the sample size is. (Any guess will do as a
starting point, although the closer to the true value your guess turns out to be, the sooner you will
be finished.)
(2) Use that value of the sampling size (i.e., your initial guess) to solve the precise formula
equation for sampling error.
(3)(a) If the value of sampling error you obtain from the formula is less than the maximum
sampling error you are willing to accept, then you should decrease your sample size and solve
the equation again.
B(ii) - 7
-------
(3)(b) If the value of sampling error you obtain from the formula is greater than the maximum
sampling error you are willing to accept, then you should increase your sample size and solve the
equation again.
(4) Continue steps (3)(a) and (3)(b) above until you arrive at the appropriate sample size for the
sampling error you are willing to accept.
The Approximate Formula for Determining Sample Size
The trial and error approach described above will always give you the best possible value for sample
size. However, the process for arriving at that value can be rather tedious. For this reason, you may
want to use an approximate formula that will give you a sampling error close to the one you would
get from the trial-and-error procedure. This approximate formula needs only to be solved once - no
repeated calculations are needed. However, you will, in most cases, obtain a larger sample size than
you would get from the trial and error procedure. That is, the approximate formula will give you a
larger sample size than you need to achieve your target sampling error.
Here is the approximate formula:
NxZ2
n =
[4 x (N - 1) x E2] + [Z2]
Where:
n = sample size
N = number of customers served (from which the sample is to be drawn)
E = the maximum acceptable level of sampling error, expressed as a decimal
fraction (e.g., 5% = 0.05)
Z = the Z-score corresponding to the confidence level selected
(this can be obtained from most standard statistics references,
including most basic statistics textbooks). The Z-scores for the
80%, 90% and 95% confidence levels are given above in this Appendix in
conjunction with the precise formula.
A Combined Approach
You can, if you wish, make use of both the approximate formula and the trial and error approach
given above. Begin by using the approximate formula to get an approximate value for the sample
B(ii) - 8
-------
size. Then use this approximate value as your first guess for sample size in the trial and error
approach, and proceed from there with the trail and error approach as above.
This combined approach will allow you to come up with the lowest possible sample size with the
least amount of effort.
Why is So Much Attention Given to Sample Size?
Much of Appendix B(i) and all of this Appendix have been devoted to considerations related to
sample size. Why, you might ask, do people spend so much time worrying about sample size?
If you use a larger sample size than you need, you will incurred greater costs and impose a greater
burden on your customers than needed.
The extra costs alone can be quite considerable. Let's consider a hypothetical telephone
survey. For each extra customer in the sample, you have to spend additional time conducting
the telephone interview, following up with those who did not answer the first time, following
up with those who did not initially agree to participate, and so on. It also means more data
to be recorded and analyzed.
The extra burden on your customers' time also can be quite large when you add up the total
time spent by all customers surveyed.
If the sample size used turns out to be greater than was needed, then the extra cost incurred and the
extra burden imposed were wasted.
On the other hand, if you use too small a sample size, then you may have greater uncertainty about
the true satisfaction of your customers (because the sampling error was so large). You were
uncertain about their degree of satisfaction before (that's why you decided to conduct the survey) and
you may now find that your level of uncertainty afterward is not much reduced. In this case, the
whole cost of conducting the survey may prove to have been wasted.
Keep in mind that any wasted time and dollars could otherwise have been used to improve the
products or services you provide to your customer. So you want to choose the smallest possible
sample size that will give you a level of sampling error that you can live with. The results should
be precise enough to give you the degree of certainty you need about: (1) the true current level of
satisfaction of your customers and (2) how their degree of satisfaction has been changing over time
— as a result of your continuing efforts to improve products and services.
B(ii) - 9
-------
Appendix C: How to Obtain Clearance for EPA Customer
Satisfaction Surveys
QUESTION 1: WHO CAN USE THE CUSTOMER SERVICE ICR?
According to OMB's Resource Manual for Customer Surveys (dated October 1993) and other
relevant guidance documents, the generic clearance shall be used for "strictly voluntary collections
of opinion information from clients that have experience with the program that is the subject of each
data collection" and precludes this option for use by regulatory agencies to survey regulated entities1
• in any situation where a respondent may perceive that a response will result in risks to his
interests through potential penalties or loss of benefits
• for collecting factual information (other than simple identifying information, where needed)
• for collecting data from the general public
QUESTION 2: HOW DO I OBTAIN APPROVAL FOR MY SURVEY, IF IT MEETS THE
CONDITIONS OUTLINED ABOVE?
Prior to initiating the survey, sponsoring programs must seek final approval from OMB. To obtain
approval, sponsoring programs must submit a clearance package consisting of a memorandum and
a copy of the survey instrument through Regulatory Information Division (RID). The memorandum
will be addressed from the program or office director to the RID Desk Officer at, Office of Policy,
(2136). The memorandum must address the following2:
• Survey title, identification of survey originator (office, point of contact, phone number)
• Description and intended purpose of the survey as it relates to EPA customers
• Methodology and use of anticipated results
• Collection schedule, follow-up plans
• Costs and burden to the Agency and respondents, and the number of respondents
The memorandum will vary in length and detail, depending on the complexity of the survey. IPS staff
will review each submission to ensure that it meets the requirements of the Paperwork Reduction Act
and any conditions of the generic approval. They may rej ect any proposed customer survey that does
JEPA interprets this to preclude any EPA surveys conducted fact finding for the purposes of regulatory
development or enforcement.
2For customer feedback forms and short questionnaires, a one-page memorandum should be sufficient.
Mail or telephone surveys making use of statistical sampling must include statistician's name/phone, and a brief
design, precision requirements, and pretests/pilot tests.
C-l
-------
not meet the criteria above. In the methodological issues, the program shall solicit Agency statistical
experts through EPA's Statistical Policy Branch or program office to make any final determinations
as to the statistical validity of the customer survey.
QUESTIONS: HOW LONG WILL THE PROCESS TAKE?
Following review within RID, RID will submit surveys and attached materials to OMB for a 10
working-day review.
WHAT ELSE SHOULD I KNOW?
Sponsoring organizations within the EPA should maintain records according to each survey schedule.
In general, survey results should be maintained for three years or until after follow-up has been
performed.
Sponsoring offices are encouraged to provide feedback to RID on the success of their surveys
(through a memo, or summary report). That information may be shared with fledgling customer
survey programs within other parts of the EPA. Feedback might include:
• response rates, follow-up strategies, important lessons related to survey design and
implementation
• general trends established from analysis of data
• changes to the organization as a result of the survey
• points of contact for questions about the survey
EXAMPLE OF BURDEN STATEMENT FOR FORMS OR SURVEY
The OMB Control Number and expiration date must appear on the front page of an OMB-approved
form or survey, or on the first screen viewed by the respondent for an on-line application. The rest
of the burden statement must be included somewhere on the form, questionnaire or other collection
of information, or in the instructions for such collection.
Explain the reasons the information is planned to be and/or has been collected, and the way such
information is planned to be and/or has been used to further the proper performance of the functions
of the agency. (See the requirements of Executive Order 12862 below for ideas.) State whether
responses are voluntary, required to obtain or retain a benefit (citing authority), or mandatory (citing
authority), and the nature and extent of confidentiality to be provided, if any (citing authority).
The following information must appear on the first page of the survey:
Form Approved OMB Control No. xxxx-xxxx. Approval expires MM/DD/YY.
C-2
-------
Public reporting burden for this collection of information is estimated to average X minutes per
response, including the time for reviewing instructions, gathering information, and completing and
reviewing the collection of information. Send comments on the Agency's need for this information,
the accuracy of the provided burden estimates, and any suggestions for reducing the burden, including
the use of automated collection techniques to the Director, OPPE Regulatory Information Division,
United States Environmental Protection Agency (Mail Code 2137), 401 M Street, SW, Washington,
D.C. 20460; and to the Office of Informati on & Regulatory Affairs, Office of Management & Budget,
725 17th Street, NW. Washington, D.C. 20503, Attention: Desk Officer for EPA. Include the EPA
ICR number and the OMB control number in any correspondence.
CUSTOMER SERVICE EXECUTIVE ORDER (12862) REQUIREMENTS
• Identify customers who are or should be receiving EPA service
Survey customers for the kind/quality of services they want, their level of satisfaction with the
services, and whether standards are set for what matters to them
• Develop, post and implement standards
• Measure results against them
• Report annually to customers on progress toward achieving standards
• Integrate customer service standards, measurement and tracking with reinvention, planning,
budgeting (GPRA), operating plans, regulations and guidelines, training and personnel
classification and evaluation
• Recognize employees for meeting and exceeding customer service standards
• Benchmark customer service performance against the best in business
• Survey front-line employees on barriers to, and ideas for, matching the best in business
• Provide customers with choices in sources of service and methods
• Make information, services and complaints systems easily available
• Address customer complaints
• Develop cross-media (within agency) and cross-Agency programs to serve shared customer
groups
Take advantage of new technology to better serve customers
C-3
-------
Following are examples of successful applications to OMB.
Sample #1
U.S. ENVIRONMENTAL PROTECTION AGENCY
REGION I
OFFICE OF ENVIRONMENTAL MEASUREMENT & EVALUATION
60 WESTVIEW STREET, LEXINGTON, MA 02173-3185
MEMORANDUM
DATE: June 12, 1997
SUBJECT: Request for OMB Approval of Customer Feedback Survey
FROM: Carol Wood, Manager
Ecosystems Assessment Branch
TO: Barbara Willis, RID Desk Officer
Regulatory Information Division
Office of Policy, Planning and Evaluation
EPA's Region 1, New England Office is preparing to distribute copies of the 1997 State of the New England Environment Report.
In order to learn whether the report is clear, easy to read and provides information that our customers need, we are preparing a
customer feedback survey to include with the report. A copy of the survey form is attached.
Approximately 12,000 copies of the report will be distributed to EPA personnel, citizens, local, state and federal offices out side the
EPA Region 1 Office, with the survey form as an insert. We expect to receive approximately 3,000 responses. Region I will create
a database to track survey form responses. The information will be used to prepare a report which will summarize the findings and
make recommendations on how to improve the next State of the New England Environment Report and our other outreach
activities.
We will be receiving the reports from the Government Printing Office by June 24 and hope to receive approval for the customer
survey form and have the forms ready to include in the mailings.
If you have any questions or concerns about this request, please contact Diane Switzer at 617-860-4377 or me at 617-860-4316.
Attachments
C-4
-------
Sample #1 -Continued
Request for Approval of Information Collection Activity
I. Background
The 1997 State of the New England Environment Report \s an outreach tool, designed to inform the public on environmental
conditions, using indicators that have been selected in the National and Regional processes as we begin focusing more on
environmental results. We discuss topics of concern to the public and EPA, signs of improvement or degradation, and what EPA
and our partners are doing to improve conditions. The purpose of this outreach activity is to provide clear and concise information
to the public that meets their informational needs and allows them to better understand what we are doing to improve and protect
the environment and public health. The discussion topics are selected based upon regional priorities and what we think the public
wants to know.
II. Survey Purpose and Description
The State of the New England Environment Workgroup is planning to conduct a customer feedback survey in the form of a
"Reader's Evaluation Form," to evaluate whether we are providing the public with the information they want and need in a way
that is easy to read and use. The results will be used to improve the report's content, readability and use.
The evaluation consists of six questions. The first question will evaluate the reports readability. The second question evaluates
how well we do in communicating information the public wants to know. The third question evaluates how the information is
useful to the reader. The fourth and fifth questions evaluate the information needs of the reader that we are not meeting. The sixth
question evaluates whether the report is something the public wants to receive.
III. Survey Methodology and Use of Results
The potential target audience for the evaluation forms consists of approximately 12,000 citizens, businesses and government
personnel (local, state and national). EPA Region I plans to distribute the forms as inserts to copies of the 7997 State of the New
England Environment Report. Through this effort, we anticipate that approximately 3,000 readers will respond. We estimate that
it will take a respondent approximately five minutes to complete an evaluation form.
EPA Region I will create a database to track evaluation form responses. The information will be used to prepare a report which
will summarize the findings and make recommendations to the State of the New England Environment Workgroup and Regional
managers on how to improve the readability, use and content of this report and other similar outreach activities.
IV. Respondents' Burden
Number of Respondents 3,000
Minutes per Response 5 minutes x 3,000 = 15,000 minutes = 250 hours
CostperHour $11.00*
Total Burden: 250 Hours; $2,750
* Based on Federal/State/Local Employment & Payroll averages as presented in the 1996 Statistical Abstract of the United States
V. Agency Burden
EPA Staff Time 100 hours
Cost per Hour $36.00
Total Burden 100 hours; $3,600.00
C-5
-------
Sample #1 -Continued
YOUR COMMENTS, PLEASE . . .
We would like to know if the 1997 State of the New England Environment Report provides you with use useful information.
Your responses to the following questions will help us meet your needs.
1. a. Is this report easy to read and understand? Yes_No_
b. What would make the report easier to read and use?
2. Please rate the report as to how informative the discussions within each of the sections are, with 1 = not informative and 5 =
very informative.
Report Section Not Informative Very Informative
a. New England Ecosystems 12345
b. Public Health & Our Environment 12345
c. Economic Opportunities 12345
d. Recreational Resources 12345
e. Environmental Education & Outreach 12345
f. New Directions 12345
3. In which areas is the report helpful to you? School Work Home
Leisure Time Local Community General Knowledge Other
4. What topic(s) would you like to see in future reports?
5. We welcome any other comments you have about this report:
6. Would you like to receive a copy of future reports? Yes No
If "Yes," please provide your mailing address:
Name
Organization_
Address
Town/City State
Zip Code County
Please fold in half with EPA's return address on the outside, staple/tape shut, and mail.
Thank you for your response! Environmental Protection Agency Region I, New England Office
C-6
-------
Sample #1- continued
June 18, 1997
MEMORANDUM
SUBJECT: Review of Customer Satisfaction Questionnaire,
ICRNo. 1711.01 (OMB 2090-0019)
FROM: Barbara N. Willis
Regulatory Information Division (2136)
To: Chris Wolz
Natural Resources, OIRA
As a condition of OMB approval for the generic ICR, EPA agreed to submit each specific questionnaire covered by this
clearance to OMB for review. Therefore I am forwarding for your review Region I "1997 State of the New England
Environmental Report". The purpose of this survey is to evaluate whether Region I is providing the public with the information
they want and need in a way that is easy to read and use. The results will be used to improve the reports' content, readability and
use.
Your comments and suggestions would be much appreciated. Thank you for your cooperation in this matter. If you have any
questions, please contact me at (202) 260-9453.
Attachments
Sample #2
MEMORANDUM
SUBJECT: Submittal of Customer Satisfaction Survey for Expedited
OMB Review
FROM: Michael B. Cook, Director
TO: Matt Leopard, RID Desk Officer
Office of Policy, Planning and Evaluation (2136)
Attached is a clearance package for an Office of Water Customer Satisfaction Survey as authorized under Executive
Order 12862, "Setting Customer Service Standards." This particular survey is designed to assess state opinion on the
current level of satisfaction and desired improvements to the Agency's Water grant process. This voluntary survey focuses
on three of the primary water quality management grants under the Clean Water Act, Sections 106, 319 and 604 (b).
We are requesting an expedited reviefor this survey instrument in order to comply with the rather tight schedule that is
mandated under the Executive Order. We anticipate initiating the survey no later than mid-November. I am requesting your
assistance in coordinating this review.
Please contact Jane Ephremides of my staff (260-5835), or Don Brady in the Office of Wetlands, Oceans and
Watersheds (260-7074) if you have any questions.
Attachment
cc:
Bob Wayland
Abby Pirnie
Don Brady
C-7
-------
Sample #2 - continued
CLEARANCE INFORMATION COLLECTION REQUEST FOR 1994 THE CUSTOMER SATISFACTION
SURVEY
Identification of Information Collection
Executive Order 12862 requires Agencies to "survey customers to determine the kind and quality of services they want and
their level of satisfaction with existing services". This survey, will be conducted by customer satisfaction survey
professionals at the request of the Environmental Protection Agency's Office of Wastewater Management' Resource
Management and Evaluation Staff and the Office of Wetlands, Oceans and Watersheds Assessment and Watershed
Protection Division. Tim Icke, Program Analyst, will be the point of contact at OWOW's Assessment and Watershed
Protection Division. He can be reached at (202)-260-2640.
Short Characterization of the Survey
The 1994 Customer Satisfaction Survey will solicit opinions from members of the grants community within the States. The
data collection is authorized by Executive Order Number 12862, "Setting Customer Service Standards," which requires all
federal executive departments and agencies that provide significant services directly to the public to carry out the principles
of the National Performance Review.
As a result of the Executive Order, The Office of Water is assessing its operations and procedures in order to provide service
to the public that matches or exceeds the best service available in the private sector. The Customer Satisfaction Survey
applies to three of the grants, those under Sections 106, 319, and 604(b) of the Clean Water Act. The survey is intended to
determine the customers' current level of satisfaction and desired improvements in these three grant programs. In the water
program, there are 11 sources of financial assistance available to assist the states and territories in achieving the mandates of
the Clean Water Act. The questions focus on respondents' opinions and perceptions of services rendered.
Collection Methodology
Using a pretested telephone questionnaire, EPA will survey State water quality managers, grants administration managers,
and the program managers for Sections 106, 319, and 604(b) in each of the 57 States and territories. EPA estimates that the
number of respondents will vary considerably from state to state. Using a conservative estimate, the highest possible burden
will be 5 respondents per state. The survey instrument is a 15-minute, voluntary telephone questionnaire covering
approximately 30 questions. There are four open-ended questions. For those customers that request an opportunity to
respond at greater length, follow-up calls will be scheduled. Since these conversations are voluntary, will vary greatly, and
will affect a small percentage of respondents, the follow-up calls are not considered burdens under the definition of the
Information Request.
This one-time only information collection will involve approximately 285 voluntary respondents of which 70% are
anticipated to complete the telephone survey. The survey will require approximately 50 hours at a total cost to the
respondents of $1,448. Exhibit I-a, Respondent Burden and Costs, provides a detailed description of the unit burden and
costs to respondents for this collection. The average burden per response is 15 minutes.
State grant program authorities are the only respondent group that will be affected by this survey, and by definition they are
not small governmental jurisdictions.
Use of Survey Results
The results of the Customer Satisfaction Survey will be summarized in a report or accompanying briefing document. EPA
intends to use the information gathered by the survey to identify tools to improve the grants management process by reducing
paperwork, focusing on results while maintaining accountability, and responding to State environmental priorities. The
fundamental purpose of the customer satisfaction survey is to assess states satisfaction with the grant process and existing
services. The survey will help EPA:
S Identify potential changes that states would like to see in the administrative management of Sections 106, 319,
and 604 (b) grant programs;
S Assess the three grant programs' potential to enhance/retard state adoption with the watershed protection
approach; and
S Understand state level of satisfaction/dissatisfaction with the three grant programs.
-------
Sample #2 - continued
Collection Schedule and Follow-up Plans
EPA seeks to minimize the amount of data Collected through a one-time only data gathering effort while at the same time
gathering enough information for an effective Customer Satisfaction Survey. The survey will help Headquarters establish a
benchmark to compare EPA's customer service performance with that of other federal agencies and private sector businesses.
In the future, this information will help to provide customers with choices in both the sources of service and the means of
delivery; to make information, services, and complaint systems easily accessible; and to provide a means to address customer
complaints.
Costs and Burden to the Agency and Respondents, and Number of Respondents
The total burden for EPA Regional and State grants program authorities is a function of the number of grants managers,
auditors and program managers for Sections 106, 319 and 604(b) of the Clean Water Act in each state and interstate agency
and the number of open-ended questions. Exhibits 1-a and 1-b give detailed descriptions of the individual reporting and
record keeping requirements associated with the survey. Burden estimates are based on EPA data from the Regions and
Headquarters.
Exhibit 1-a summarizes the state respondents' burden and costs as respondents to the voluntary telephone survey. The total
respondent burden associated with the Customer Satisfaction Survey is 50 hours (200 respondents at 15 minutes per call)
and the total respondent cost is $1,448, which equates to a cost per respondent of $7.24. This estimate assumes that the
average hourly labor cost for state employees is $28.96, comparable to a GS9, Step 10 salary.
The Agency's burden and cost arises from contacting appropriate regional program officers, and from reviewing, analyzing,
and processing the data. The total annual Agency burden associated with the customer Satisfaction Survey is 100 hours.
This assumes that the average hourly labor cost of federal employees is $28.96, equal to a GS-9, Step 10 salary. The total
annual Agency cost resulting from survey reporting and record keeping resulting from the customer survey is $2,896.
C-9
-------
Exhibit l-a
Respondent Burden and Costs
Regulation Requirements
Survey Reporting
requirements (one-time
only)
Respond to telephone
Customer Satisfaction
Survey
Total Burden and Costs
for all affected
Respondents:4
(A)
Total #
Respondents1
285
(B)
#
Responses2
200
(C)
Composite
hrs/respondent
0.25
(D)
Total hours
(B)*(C)
50
50
(E)
Hourly
labor .costs3
$28.96
(F)
Total
Cost (D)*(E)
$1,448
$1,448
1 Respondents include State grants managers, auditors, and program managers for
Sections 106, 319, and 604 (b) in each of the 57 States and Territories.
2 Assumes approximately five calls to each State and Territory and assumes 70%
response rate.
3 Hourly labor cost equals the annual salary for a GS-9 step 10 (37,651) times 1.6 (the
benefits multiplication factor as listed in the June 1992 ICR Handbook) and divided by 2,080 of
work hours per year).
4 Numbers may not add due to rounding.
C- 10
-------
Exhibit 1-b
Agency Burden and Costs (As users of Data)
Regulation
Requirements
Recordkeeping
Requirements
(Ongoing)
Agency
Reviews 1st
Draft of Report
Agency
Approves Final
Draft of Report
Total Agency
Burden and
Costs:2
(A)
Total no. of
respondents
N/A
N/A
(B)
No. of
responses
N/A
N/A
(C)
Composite
hours per
respondent
N/A
N/A
(D)
Total hours
(B) * (C)
60
40
100
(E)
Hourly labor
cost1
$28.96
$28.96
(F)
Total Costs
(D)*(E)
$1,738
$1,158
$2,896
IHourly labor cost equals the annual salary for a GS-9 step 10 ($37,651) times 1.6 (the benefits multiplication factor as listed in the June 1992 ICR
Handbook) and divided by 2,080 of work hours per year).
2 Numbers may not add due to rounding.
C-ll
-------
Sample # 2 - continued
Draft - October 25, 1994
1994 CUSTOMER SATISFACTION SURVEY
HOW ARE WE DOING?
Grant Administration: Grant Administration Staff
Only a sample of the several versions of the surveys for a series of grants is presented.
Hello, may I please speak with (NAME FROM FACE SHEET)?
RESPONDENT AVAILABLE
RESPONDENT NOT AVAILABLE (SCHEDULE A CALL BACK)
Hello, my name is Of Abt Associates. We are conducting a customer
satisfaction study for the Environmental Protection Agency (EPA) about three Office of Water program management grant
programs. The study is voluntary and the answers that you give will be kept strictly confidential.
1) Are you familiar with the Section 106 grant program that funds the management of state water quality programs?
YES 1
NO (SKIP TO QUESTION 14) 2
DO/REF (SKIP TO QUESTION 14) 3
2) I have some questions about the FY 94 grant cycle. How satisfied are you with the level of reporting burden
under Section 106? Are you...
Very satisfied (SKIP TO QUESTION 4) 1
Satisfied (SKIP TO QUESTION 4) 2
Dissatisfied 3
Very dissatisfied 4
3) What are the one or two most important changes you would like to see in Section 106 reporting requirements?
4) Do you think EPA made good use of the FY 94 section 106 data you reported to them?
Yes 1
NO 2
Grants Administrators 1
5) Were any of the reports created by your state in complying with Section 106 requirements for FY 94 useful for
other state purposes such as state budgeting or accounting?
Yes 1
NO 2
6) How satisfied are you with the opportunity offered by EPA to file Section 106 FY 94 reports electronically? Are
you...
Very satisfied (SKIP TO QUESTION 8) 1
Satisfied (SKIP TO QUESTION 8) 2
Dissatisfied 3
Very dissatisfied 4
C- 12
-------
7) What are the one or two most important changes you would like to see in Section 106 electronic reporting scope
or procedures?
8) How satisfied were you with the length of time it took EPA to respond to requests for information on grant
administration and reporting for FY 94 Section 106 grants? Were you...
Very satisfied 1
Satisfied 2
Dissatisfied 3
Very dissatisfied, or 4
Did you not make any requests for information 5
9) How satisfied are you with the length of time it took to obtain the EPA approvals required at various states of
administration of FY 94 Section 106 grants? Were you...
Very satisfied 1
Satisfied 2
Dissatisfied 3
Very dissatisfied, or 4
Did you not need any EPA approvals 5
10) How satisfied are you with EPA's requirements for the close-out or rollover of the Section 106 grant fund? Are
you...
Very satisfied 1
Satisfied 2
Dissatisfied 3
Very dissatisfied 4
Grants Administrators 2
11) What are the one or two changes you would most like EPA to make in its Section 106
reporting requirements?
12) Overall, how satisfied are you with EPA's FY '95 Section 106 grant programs? Are you...
Very satisfied 1
Satisfied 2
Dissatisfied 3
Very dissatisfied 4
13) What is the one most important change you would like to see made to the Section 106 grant program?
Grants Administrators 3
REPEAT FOR SECTION 319 AND 604(B)
INSERT THE FOLLOWING AFTER QUESTION 1 FOR 605(b)
14) In meeting the reporting requirements for the Section 106. 319 and 604(b) programs for FY94, did your state
ever have to submit the same report the different grant programs?
YES 1
No 2
15) Please compare your state's experience with the section 106, 319, and 604b programs for FY 95 with that of
other grant programs administered by the EPA Office of Water.
CLOSING: Thank you very much for your time. Analysts working on the project may contact you later for further
detail or clarification of the information you've given. Is there a best time of day or day of the week to reach
you?
Thanks again, Goodbye.
C-13
-------
OMB Control No. XXXX-XXX
U.S. ENVIRONMENTAL PROTECTION AGENCY
APPLICANT PERMITTING SURVEY
Introduction:
The attached survey is a follow-up to your recent permit application (or modification request)
with the US EPA. We are interested in improving our permitting system, and we recognize that
to do so, we need your frank input.
Problem areas that are identified will be followed up with focus groups to obtain more specific
insights. While participation in this survey is voluntary, we encourage you to take this
opportunity to help us improve the quality of our permitting processes.1
Instructions:
Please complete this survey by circling your answers and returning it in the postage-paid envelope
provided. Most of the questions in this survey ask that you rate some aspect of US EPA's
performance on the following scale: "1" means you are very dissatisfied, and "6" means you
are very satisfied. If a question does not apply to your interaction with US EPA, please skip it
and go on to the next question.
This survey is estimated to take an average of 10 minutes to complete.
The public reporting and recordkeeping burden for this collection of information is estimated to average
15 minutes per response annually. Burden means the total time, effort, or financial resources expended
by persons to generate, maintain, retain, or disclose or provide information to or for a Federal Agency.
This includes the time needed to review instructions; develop, acquire, install, and utilize technology
and systems for the purposes of collecting, validating, and verifying information, processing and
maintaining information, and disclosing and providing information; adjust the existing ways to comply
with any previously applicable instructions and requirements; train personnel to be able to respond to a
collection of information; search data sources; complete and review the collection of information; and
transmit or otherwise disclose the information. An agency may not conduct or sponsor, and a person is
not required to respond to, a collection of information unless it displays a currently valid OMB control
number.
Send comments on the Agency's need for this information, the accuracy of the provided burden estimates, and any
suggested methods for minimizing respondent burden, including through the use of automated
collection techniques to the Director, OPPE Regulatory Information Division, U.S. Environmental
Protection Agency (2136), 401 M Street, S.W., Washington, D.C. 20460. Include the OMB control number in
any correspondence. Do not send the completed survey to this address.
D- 1
-------
U.S. ENVIRONMENTAL PROTECTION AGENCY
APPLICANT PERMITTING SURVEY
Very Ver y
Dissatisfied Satisfied
123456
1) Please identify to which permit media program your response to this survey applies: (you may checkmore
than one, as appropriate, or, if your responses would appreciably differ for different program areas, please
reproduce this form and submit one for each program area.)
a) Air ( )
b) Water ( )
c) Hazardous Waste ( )
d) Other ( ). Please specify:
2) Pre-application meeting/discussion: These questions cover the pre-application discussion or meeting
(i.e., a phone call or meeting) with US EPA to discuss the application process before you submitted the
application.
a) How satisfied are you with the availability of US EPA staff
responding to your pre-application questions? 123456
b) How satisfied are you with the assistance provided by US EPA
staff during the pre-application meeting/discussion? 123456
c) How satisfied are you with the usefulness of the information
provided to you through the pre-application
meeting/discussion? 123456
d) How satisfied are you that the US EPA staff provided
suggestions or information to help minimize the overall
permitting burden (e.g., as using pollution prevention
opportunities to reduce emissions or identifying future needs
now to minimize the need for modifications later)? 123456
3) Permit Application Review and Determination: These questions cover the time period from the
submission of your permit application to US EPA's decision to either issue or deny the permit
a) How satisfied are you with the clarity of the permit application
forms? 123456
b) How satisfied are you with the clarity of the accompanying
instructions or guidance? 123456
c) How satisfied are you with US EPA's timeliness in notifying
that your application was complete? 123456
D-2
-------
d) If you received any requests for supplemental
information by the US EPA, how satisfied are you in the
following areas?
1) clarity 123456
2) timeliness of US EP A's request 123456
3) relevance 123456
e) How satisfied are you with US EPA's timeliness in determining
the issuance or denial of your permit? 123456
f) How satisfied are you with the clarity of the final permit
decision? 123456
4) Overall satisfaction: These questions cover your overall level of satisfaction with the manner which the
permit process was handled by US EPA.
a) Overall, how satisfied are you with the way the permitting
process was managed? 123456
b) Overall, how satisfied are you that the US EPA permitting
staff treated you in a courteous manner? 123456
c) Overall, how satisfied are you with the quality and timeliness
of the communications you have received from US EPA?
123456
d) Overall, how satisfied are you that the US EPA permitting
staff respond to your needs for guidance, information, or
technical support under the permit process? 123456
5) Would you like someone with the US EPA to contact you regarding this survey?
Yes Please complete question 6
No Please complete question 6 (optional). Your responses will be used by US EPA for
informational purpose only.
6) Please provide the following information:
Name:
Organization:
Address:
Town/City: State.
Zip Code:
Telephone Number:( )
7) Please provide any other comments you would like us to consider:.
Thank you for taking the time to complete this survey.
D-3
-------
OMB Control No. XXXX-XXX
U.S. ENVIRONMENTAL PROTECTION AGENCY
CITIZENS PERMITTING SURVEY
Introduction:
The attached survey is a follow-up to your recent participation in a US EPA permitting action, through
public comment or attendance at a US EPA hearing or meeting. We are interested in improving our
permitting processes, and we recognize that to do so, we need your frank input.
Problem areas that are identified through the use of this survey will be followed up with focus groups to
obtain more specific insights. While participation in this survey is voluntary, we encourage you to take
this opportunity to help us improve the quality of our permitting process.1
Instructions:
Please complete this survey by circling your answers and returning it in the postage-paid envelope
provided. Most of the questions in this survey ask that you rate some aspect of US EPA's performance
on the following scale: "1" means you are very dissatisfied, and "6" means you are very satisfied.
If a question is not applicable to your interaction with EPA, please skip it and go on to the next question.
This survey is estimated to take an average of 10 minutes to complete.
The public reporting and recordkeeping burden for this collection of information is estimated to average
10 minutes per response annually. Burden means the total time, effort, or financial resources expended
by persons to generate, maintain, retain, or disclose or provide information to or for a Federal Agency.
This includes the time needed to review instructions; develop, acquire, install, and utilize technology
and systems for the purposes of collecting, validating, and verifying information, processing and
maintaining information, and disclosing and providing information; adjust the existing ways to comply
with any previously applicable instructions and requirements; train personnel to be able to respond to a
collection of information; search data sources; complete and review the collection of information; and
transmit or otherwise disclose the information. An agency may not conduct or sponsor, and a person is
not required to respond to, a collection of information unless it displays a currently valid OMB control
number.
Send comments on the Agency's need for this information, the accuracy of the provided burden estimates, and any
suggested methods for minimizing respondent burden, including through the use of automated
collection techniques to the Director, OPPE Regulatory Information Division, U.S. Environmental Protection
Agency (2136), 401 M Street, S.W., Washington, D.C. 20460. Include the OMB control number in any
correspondence. Do not send the completed survey to this address.
D-4
-------
U.S. ENVIRONMENTAL PROTECTION AGENCY
CITIZENS PERMITTING SURVEY
Very Very
Dissatisfied Satisfied
1234 56
1) Please identify to which permit media program your response to this survey applies: (you may checkmore
than one, as appropriate, or, if your responses would appreciably differ for different program areas, please
reproduce this form and submit one for each program area.)
a) Air ( )
b) Water ( )
c) Hazardous Waste ( )
d) Other ( ). Please specify:
2) How satisfied are you with US EPA's notification to provide the
public with the opportunity to comment in a timely manner? 123456
3) How satisfied are you with US EPA's method of notification to
provide the public with the opportunity to comment in a
convenient manner (e.g., by newspaper, radio, direct mailing or
other preferred means)? 123456
4) How satisfied are you that US EPA provided clear and concise
information about the permit process and application? 123456
5) How satisfied are you with the suitability of the public
hearing/meeting location? 123456
6) How satisfied are you with the suitability of public
hearing/meeting time? 123456
7) How satisfied are you that US EPA provided you opportunities
to present your comments? 123456
8) How satisfied are you that US EPA presented the criteria that
will be used for the permit decision? 123456
9) How satisfied are you with the quality of US EPA's response to
your written or oral comment? 123456
10) Overall satisfaction: These questions cover your overall level of satisfaction with the manner which the
permit process was handled by US EPA.
a) Overall, how satisfied are you with the way the permitting
process was managed? 123456
b) Overall, how satisfied are you that the US EPA permitting
staff treated you in a courteous manner? 123456
D - 5 SAMPLE - Not Approved by OMB
-------
c) Overall, how satisfied are you with the quality and timeliness
of the communications you have received from US EPA?
123456
d) Overall, how satisfied are you that the US EPA permitting
staff respond to your needs for guidance, information, or
technical support under the permit process? 123456
11) Would you like someone with the US EPA to contact you regarding this survey?
Yes Please complete question 12
No Please complete question 12 (optional). Your responses will be used by US EPA for
informational purpose only.
12) Please provide the following information:
Name:
Organization:
Address:
Town/City: State
Zip Code:
Telephone Number:
13) Please provide any other comments you would like us to consider, including changes to the permitting
process:
14) Please put an "x" next to the line in each category that best describes you:
I live near the facility/facilities requesting the permit(s).
I am employed by the facility requesting the permit.
I am employed by the local or state government in which the facility is located.
I am a member of a local environmental or community group.
I am a member of a regional or national environmental or community organization.
Other (Please specify)
Thank you for taking the time to complete this survey.
D - 6 SAMPLE - Not Approved by OMB
-------
OMB Control No. xxxx-xxxx
U.S. ENVIRONMENTAL PROTECTION AGENCY
DELEGATED AUTHORITY PERMITTING SURVEY
Introduction:
The attached survey is being distributed to state, tribal and/or local governments that have been delegated
or authorized to administer a permitting program under federal statutes administered by US EPA. We
are interested in improving our permitting delegation/authorization and evaluation processes, and we
recognize that to do so, we need your frank input.
Problem areas that are identified will be followed up with focus groups so that we may obtain more
specific insights. Focus groups will include state and regional personnel and other stakeholders as
appropriate. While participation in this survey is voluntary, we encourage you to take this opportunity
to help us work better for all of our stakeholders.1
Instructions:
Please complete this survey by circling your answers and returning it in the postage-paid envelope
provided. Most of the questions in this survey ask that you rate some aspect of US EPA's performance
on the following scale: "1" means you are very dissatisfied, and "6" means you are very satisfied.
If a question does not apply to your interaction with US EPA, please skip it and go on to the next
question.
This survey is estimated to take an average of 15 minutes to complete.
The public reporting and recordkeeping burden for this collection of information is estimated to average
10 minutes per response annually. Burden means the total time, effort, or financial resources expended
by persons to generate, maintain, retain, or disclose or provide information to or for a Federal Agency.
This includes the time needed to review instructions; develop, acquire, install, and utilize technology
and systems for the purposes of collecting, validating, and verifying information, processing and
maintaining information, and disclosing and providing information; adjust the existing ways to comply
with any previously applicable instructions and requirements; train personnel to be able to respond to a
collection of information; search data sources; complete and review the collection of information; and
transmit or otherwise disclose the information. An agency may not conduct or sponsor, and a person is
not required to respond to, a collection of information unless it displays a currently valid OMB control
number.
Send comments on the Agency's need for this information, the accuracy of the provided burden estimates, and any
suggested methods for minimizing respondent burden, including through the use of automated collection
techniques to the Director, OPPE Regulatory Information Division, U.S. Environmental Protection Agency
(2136), 401 M Street, S.W., Washington, D.C. 20460. Include the OMB control number in any correspondence.
Do not send the completed survey to this address.
D-7
-------
DELEGATED AUTHORITY SURVEY
Very Very
Dissatisfied Satisfied
1 2345 6
1) Please identify to which permit media program your response to this survey applies: (you may
check more than one, as appropriate, or, if your responses would appreciably differ for
different program areas, please reproduce this form and submit one for each program area.)
a) Air ( )
b) Water ( )
c) Hazardous Waste ( )
d) Other ( ). Please specify:
2) Planning & Priority-Setting: These questions cover your opinion of our planning and
priority-setting prior to finalizing each annual permits program work plan or performance
agreement. Please rate our performance on the following criteria:
a) How satisfied are you with US EPA conducting joint
planning and priority-setting with you as an equal partner: 123456
b) How satisfied are you with US EPA addressing
state/tribal/local needs and specific circumstances in final
agreements: 123456
c) How satisfied are you with US EPA identifying areas where
Federal technical assistance is needed: 123456
d) How satisfied are you with US EPA's developing clear
program elements (i.e., those that are mandatory for the
delegated/authorized program): 123456
e) How satisfied are you with US EPA's developing a permit
delegation/authorization that is aligned with your state, tribal
or local environmental strategic plan: 123456
3) Technical Assistance: These questions ask for your opinion of our performance in providing
technical assistance to you, based on the following criteria:
a) How satisfied are you with the quality of the training we have
provided to you: 123456
b) How satisfied are you with the accuracy of our answers to
your technical questions: 123456
c) How satisfied are you with the timeliness of our answers to
your technical questions: 123456
d) How satisfied are you with our willingness to work with you
creatively to solve difficult permitting problems: 123456
D - 8 SAMPLE - Not Approved by OMB
-------
4) Oversight: These questions ask for your opinion of the Federal oversight conducted on
individual permits:
a) How satisfied are you that US EPA is conducting oversight
consistent with the agreement, if an agreement is in place: 123456
b) How satisfied are you that US EPA is providing an
appropriate amount of oversight: 123456
c) How satisfied are you with the clarity of the comments that
were made: 123456
d) How satisfied are you with the appropriateness of the
comments: 123456
e) How satisfied are you with the timeliness of the comments: 123456
f) How satisfied are you with US EPA's ability to provide
comments which were helpful in maintaining or improving
the quality of permit decisions: 123456
g) How satisfied are you with resolving issues that we have
raised: 123456
h) If you were to recommend an improvement in the agreement, it would be
that:
We should review fewer permits on a real-time basis
We should review more permits on a real-time basis
We should review only specific types of permits
(If yes, which types?)
We should conduct permit reviews to help implement new programs
We should review permits only when requested by the authorized entity or
other stakeholder
5) Permitting Program Evaluation: These questions concern our evaluation of your permitting
program. Please rate our performance on the following:
a) How satisfied are you that the delegation/authorization
agreement meets your agency's permitting needs: 123456
b) How satisfied are you with our evaluation of your
delegated/authorized permitting program in terms of:
1. Accuracy: 123456
2. Identification of accomplishments: 123456
3. Identification of opportunities for improvement: 123456
4. Resolution of disagreements: 123456
5. Suggestions for program improvements: 123456
6. Your overall performance: 123456
D - 9 SAMPLE - Not Approved by OMB
-------
6) Keeping you informed: This question covers our communication with you regarding real or
potential changes to permitting regulations, process or delegations/ authorizations, and
providing you the opportunity to comment on them. Please rate our performance on the
following criteria:
a) How satisfied are you with the timeliness of US EPA's
notification of the opportunity to comment: 123456
b) How satisfied are you with the convenience of the method of
comment (e.g., in writing, through meetings, etc.): 123456
c) How satisfied are you with US EPA providing information
about real or potential changes to the permitting relationship
in an understandable manner: 123456
d) How satisfied are you with our responsiveness to your
comments: 123456
7) Trust and Fairness: These questions concern your opinion of how well we manage the
delegation of authority:
a) How satisfied are you with the timeliness of US EPA's
responsiveness to your request for delegation/authorization: 123456
b) How satisfied are you with US EPA providing valuable
review and comments on your request for
delegation/authorization: 123456
8) If delegation/authorization occurred in the last 3 years, How
satisfied are you with the delegation process overall: 123456
9) Please provide the following information:
Name:
Organization:
Address:
Town/City: State
Zip Code:
Telephone Number:
10) Please provide any other comments you would like us to consider:.
D - 10 SAMPLE - Not Approved by OMB
------- |