United State*
Environmental Protection
Agency
Office of Policy,
Planning, and Evaluation
Waihington, DC 20460
EPA-230/12-84-002
November 1983
Survey Management
Handbook
Volume I:
Guidelines for
Planning and Managing a
Statistical Survey
-------
For additional copies, please contact:
N. PHILLIP ROSS, Chief,
Statistical Policy Branch,
Office of Standards and Regulations
U.S. Environmental Protection Agency
PM-223, 401 M Street, S.W.,
Washington, D.C. 20460
Originally published 1983
Revised printing: November 198A
SURVEY MANAGEMENT HANDBOOK STAFF
Project Manager
MEL KOLLANDER, EPA
Principal Writer ..
Statistical Advisor
CYNTHIA CROCE, Consultant
DR. LEONARD LOSCIUTO,
Director, Institute
for Survey Research,
Temple University
Editor and Proofreader ..
PATRICIA MINAMI, EPA
-------
ENVIRONMENTAL PROTECTION AGENCY
SURVEY MANAGEMENT HANDBOOK
Volume I:
Guidelines for Planning
and Managing a Statistical Survey
Prepared by:
Statistical Policy Branch
Chemicals and Statistical Policy Division
Office of Standards and Regulations
Office of Policy, Planning, and Evaluation
U. S. Environmental Protection Agency
401 M Street, S.W.
Washington, D.C. 20460
-------
FOREWORD
WHY THIS HANDBOOK?
The purposes of this handbook are to
Improve the accuracy and usefulness of EPA
survey data and statistics;
Help EPA project management plan and manage
a contract survey; and
Guide staff through the maze of Federal and
Agency requirements normally encountered in
developing a statistical survey.
Statistical surveys are fundamental to the effective management
of many of the regulatory and enforcement programs EPA uses to
fulfill its mandate to protect the Nation's health and the
environment. EPA uses surveys not only to measure the effec-
tiveness of its regulatory and enforcement activities in gener-
al, but also to measure public concern about the standards and
regulations it is considering, to understand better the public's
behavior and attitudes regarding key environmental problems, to
measure the health risks of various substances and the health
status of various populations, to characterize public attitudes
towards proposed or existing rules, and to measure the effects
of its rules on particular marketplaces.
Most Agency surveys are national or at least regional in scope,
and therefore involve the collection and processing of large
quantities of technical data. Since EPA has limited statistical
and technical staff for processing major information collec-
tions, Agency offices often retain qualified research firms in
the private sector to collect and process survey data. These
firms typically have a national network of trained interviewers
and sophisticated information management capabilities.
Survey support services may be obtained under several procure-
ment options -- competitive arrangements, grants, cooperative
agreements, sole-source contracts, and Small Business set-
asides. However, our focus in this handbook is on open, com-
petitive, negotiated procurements. Competitive procurements
i
-------
offer the sponsoring office the maximum opportunity to establish
the survey design specifications, set standards for contractor
performance, and select firms offering the best combination of
technical quality and reasonable cost. In all cases, of course,
the sponsoring office is fully responsible for the outcome of
the project.
WHAT IS COVERED?
This first volume of the Survey Management Handbook is organized
into eight chapters, which examine the most important steps in
designing a survey and procuring the services of a competent
contractor to develop the design and carry out the field work.
It is addressed to the project officer, the EPA program official
delegated responsibility for overseeing the survey.
Chapter I discusses the reasons statistical surveys are vital
to EPA's rulemaking and enforcement activities, the major
types of surveys, the various procurement options for obtain-
ing outside technical support and the roles of the principal
survey participants.
m Chapter 2 shows you how to determine whether the proposed
survey project is both necessary and feasible.
9 Chapter 3 gives instructions for developing a sound, realis-
tic survey plan.
Chapter 4 demonstrates how to initiate a competitive procure-
ment for outside technical support to collect and process
the survey data.
ฎ Chapter 5 describes how to draft a statement of work for a
typical contract survey.
ฎ Chapter 6 examines the project officer's role in the selec-
tion of the contractor, with special emphasis on how to
evaluate the technical proposals submitted in response to
the Agency's request for proposals.
9 Chapter 7 discusses the project officer's oversight respon-
sibilities after the contractor is hired.
Chapter 8 examines the project officer's role in overseeing
the documentation and dissemination of the survey findings.
Volume II, also available from our Statistical Policy Branch,
examines the methods, procedures, and quality-assurance techni-
ques typically used to collect, process, and analyze survey
data. In addition, it expands on the EPA project officer's
role in monitoring the technical progress of a survey contract.
-------
Keep in mind that statistical and information management experts
in your office or in the Office of Standards and Regulations
(OSR) are available to help with any specific problems you may
encounter during the course of your project. The Statistical
Policy Branch of OSR has a team of experts ready to assist you
in all aspects of survey design and management.
Milton Russell
Assistant Administrator
Office of Policy, Planning,
and Evaluation
iii
-------
TABLE OF CONTENTS
Page
FOREWORD
Why This Handbook? i
What Is Covered? ii
TABLE OF CONTENTS v
TABLE OF EXHIBITS ix
CHAPTER 1 - SO YOU WANT TO DO A SURVEY ...
A. Why Surveys Are Important to the Agency ... 1
B. Major Types of Surveys 3
C. Contracting Options 4
D. The Principal Participants . 5
1. Role of the Project Officer 6
2. Role of the Contracting Officer 6
3. Role of Other Agency Specialists 9
4. Role of the Contractor 9
CHAPTER 2 - HOW TO DECIDE WHETHER TO DO THE SURVEY
A. Is the Survey Necessary? 11
B. Is the Survey Feasible? 12
CHAPTER 3 - PLANNING THE SURVEY
A. Establishing the Survey Design Requirements . 15
1. Minimum Design Specifications 16
2. Optional Specifications ... 22
B. Scheduling Considerations 26
1. Preparing the Schedule 26
2. Major Administrative Time Constraints . . 28
3. Phasing the Survey 30
C. Budgeting Considerations 31
-v-
-------
Page
CHAPTER 4 - PROCURING OUTSIDE TECHNICAL ASSISTANCE
A. Initiating the Procurement -- A Team Effort . 33
B. Preparing the Procurement Request Package . . 35
1. Checklist Requirements 37
2. Supporting Documents 39
3. Competitive Evaluation Criteria 43
CHAPTER 5 - PREPARING THE STATEMENT OF WORK
A. Importance of a Well-Defined Work Statement . 45
B. Suggested Content of the Statement of Work . 47
1. Purpose, Objectives, and Background of
the Study 47
2. Survey Design Specifications 49
3. Data Collection Requirements 53
4. Data Processing, Tabulation, and
Analysis Requirements .......... 57
C. Tips on Compiling the Work Statement .... 58
CHAPTER 6 - SELECTING THE CONTRACTOR
A. The Project Officer's Role in Selecting
the Contractor 61
1. Safeguarding Information from Offerors . . 62
2. Evaluating and Scoring the Proposals ... 62
3. Supporting the Discussions and Negotiations
with Offerors in the Competitive Range . . 66
B. Evaluating the Technical Proposals 67
1. General Evaluation Considerations .... 67
2. Evaluating the Key Design Elements .... 68
3. Evaluating Key Aspects of the Work Plan . 72
4. Evaluating the Offerors' Organizational
and Staff Capabilities 76
CHAPTER 7 - MANAGING THE CONTRACT
A. Fulfilling the Agency's Contractual
Responsibilities 79
- vi i-
-------
Page
CHAPTER 7 (continued)
B. Obtaining OMB Clearances 80
C. Monitoring the Contractor's Performance ... 84
1. Monitoring Mechanisms 84
2. Monitoring the Data Collection and
Data Processing 86
D. Providing Administrative Support 90
1. Maintaining Records of the Survey .... 90
2. Reporting Contractor Progress 91
3. Authorizing Contractor Payments 91
4. Recommending Approval of Subcontracts . . 92
5. Evaluating Personnel Changes 92
6. Guarding Against Cost Overruns 92
7. Recommending Contract Modifications ... 93
8. Evaluating Contractor Performance .... 93
9. Helping to Close Out the Contract .... 93
CHAPTER 8 - REPORTING AND DISSEMINATING THE FINDINGS
A. Overseeing Preparation of the Final Report . 95
1. Focusing the Content 96
2. Organizing the Text 96
B. Circulating Drafts for Review and Comment . . 100
C. Releasing and Disseminating Survey
Data to the Public 101
1 . Procedures for Releasing Survey Data . . 101
2. Issuing Press Notices 102
3. Holding Press Conferences 102
ACRONYMS AND ABBREVIATIONS 105
APPENDICES
A - Excerpts from an OMB Clearance Package
B - Example of an EPA Survey Questionnaire
C - Example of a Telephone Interviewer's Guide
D - Standard Procurement Leadtirnes for Contract Awards
E - Procurement Request Rationale Checklist
F - Procurement Request Approvals Requirements
- ix-
-------
TABLE OF EXHIBITS
Exhibit No. Pages
1 The Project Officer's Responsibilities
During a Typical Contract Survey 7
2 "Help" Directory 8
3 Procurement Request/Order (EPA Form 1900-8) 36
4 List of Typical Survey Deliverables .... 40
5 Topic Guide for the Statement of Work ... 48
6 Proposal Technical Evaluation
(EPA Form 1900-61 ) 64-65
7 Request for 0MB Review (Standard Form 83) . 82-83
8 Example of a Cost Status Report 87
9 Suggested Content for a Survey Report . . . 97-98
-xi-
-------
CHAPTER 1
SO YOU WANT TO DO A SURVEY ...
A survey is a method or set of techniques for gathering informa-
tion from a number of people -- a "sample" -- in order to learn
something about the entire population from which the sample is
drawn. Depending on the purpose of the study, the sample may
consist of individuals or some larger unit, such as households,
farms, or businesses. The sample is not selected haphazardly
but is carefully constructed to ensure that each individual
(or unit) in the population has a known chance of being selected
to represent the entire population. In this way, the results
can be reliably projected to the larger population.
The information is collected by means of a standardized ques-
tionnaire. Individual respondents are never identified. The
information on the questionnaires is then tallied and analyzed,
and the results are presented in summary form on statistical
tables, charts, or graphs.
The remainder of this chapter discusses --
The reasons statistical surveys are vital to
EPA's rule-making and enforcement activities;
The major types of surveys;
0 The procurement options available for obtain-
ing outside technical support; and
The roles of the principal participants in an
Agency-sponsored contract survey.
A. WHY SURVEYS ARE IMPORTANT TO THE AGENCY
There are many reasons EPA needs survey data to fulfill its
Mission. At any stage in the development or implementation
of a rule, EPA may need up-to-date, accurate information
about the people that may be, or already are affected by
its regulatory efforts. Often the best way to get this
information is to go directly to the "population of inter-
est" or the "target population" and measure their opinions,
attitudes, beliefs, or perceptions about the issues the
Agency is studying.
-1-
-------
Most EPA surveys are national In scope or at least Involve
large segments of the public. To collect data from all
the people in the study population would be prohibitively
expensive and time-consuming. By interviewing a carefully
selected sample of the "target population" and using scien-
tifically valid data collection and analysis techniques,
the Agency can obtain reliable statistics. (Of course, EPA
does statistical studies of non-human populations, but sur-
veys of this kind are beyond the scope of this handbook.)
EPA frequently uses surveys to --
- Measure public concern about standards and regulations
the Agency is considering.
For example, before setting standards for the use or
manufacture of substances suspected of having an ad-
verse effect on human health, the Agency regularly col-
lects first-hand information from people living in com-
munities exposed to these substances. The statistical
information on their activity patterns, work habits,
and health histories is then combined with other tech-
nical data to give the Agency a more comprehensive
understanding of the problem.
= Better understand public behavior and attitudes.
For example, the Agency conducted a public opinion sur-
vey in Region VIII to determine local attitudes toward
air pollution -- specifically, the willingness of area
residents to comply voluntarily with the stringent
provisions in existing State Implementation Plans.
And, when "fuel switching" was of critical concern
following the last oil crisis, EPA surveyed motorists
to find out why they were using leaded gasoline in ve-
hicles designed to operate with unleaded fuel. Since
leaded gasoline adds more harmful pollutants to the
atmosphere, it was important for the Agency to gain a
better understanding of this purchasing pattern as a
basis for developing a method of minimizing fuel
switching.
= Characterize public attitudes toward a proposed or
existing rule.
Executive Order 12291 requires EPA to prepare a Regu-
latory Impact Analysis (RIA) for major regulations at
both the proposed and final rule stages. Cost-benefit
measurements are an essential part of every RIA. To
quantify the "benefits" of some proposed rules, EPA
-2-
-------
has used statistical surveys to measure the public's
"willingness to pay" to correct the environmental
problems at issue.
Measure the impact of a regulation on a particular
marketplace.
For some RIAs, EPA has used survey data to measure the
probable "costs" of regulating certain businesses or
industries. For example, the Effluent Guidelines Divi-
sion of the Office of Water has conducted statistical
surveys to help set standards for industries that dis-
charge wastewater containing specific pollutants.
Measure the effectiveness of Agency programs.
EPA frequently does surveys to collect scientific data
for measuring the effectiveness of its regulatory ef-
forts. These surveys often measure the health risks
of various pollutants or the health status of popula-
tions suspected of being "at risk."
The Agency also uses surveys to measure the usefulness
of the public information services it sponsors through
State governments, contractors, universities, etc.
Measure the effectiveness of enforcement activities.
For example, to learn more about the nature and effec-
tiveness of enforcement activities related to PCBs
(polychlorinated biphenyls), one of the enforcement
divisions recently surveyed a sample of compliance
inspections made by the Regional Offices.
B. MAJOR TYPES OF SURVEYS
Surveys are generally classified according to the method
or techniques used to collect the data. The three tradi-
tional data collection methods are (a) face-to-face inter-
views, where skilled interviewers pose questions directly
to the respondents, usually on a one-to-one basis; (b)
telephone interviews; and (c) mail questionnaires, where
the respondents complete a printed questionnaire without
supervision. Often a combination of methods is used,
particularly for a large survey, in which case the survey
is generally classified by the method initially used to
gather the information.
To explore emerging problems, the Agency is increasingly
using less structured interviewing techniques such as
-3-
-------
focus groups or in-depth invldividual interviews with a
few individuals knowledgeable about or involved in the
issues the Agency is investigating.
The method an EPA survey statistician will choose to
collect data for a particular survey depends on factors
such as
= The characteristics of the population the Agency is
studying;
= The total number of interviews needed to properly
analyze the data (called the "target response rate");
= The nature, extent, and complexity of the data;
= The respondent's obligation to reply -- sometimes EPA
can require a mandatory response, other times response
to the survey is voluntary;
= How soon the data are needed; and
How much money is available to do the survey.
(See Chapter 2 of Volume II for an analysis of the relative
merits of the most frequently used collection methods and
the factors affecting the choice of methods.)
CONTRACTING OPTIONS
As noted earlier, our focus in this handbook is on statis-
tical surveys conducted under a competitive contract
award. A competitive procurement offers the Agency the
maximum opportunity to --
= Establish the design requirements of the survey;
= Set standards for contractor performance; and
= Select the organization appearing to offer the best
combination of technical quality and cost effectiveness.
Moreover, according to 41 U.S.C. 253, an Agency contracting
officer (with certain limited exceptions) must obtain full
and open competition in soliciting offers and awarding
contracts.
Other contracting options sometimes used to obtain survey
support services are --
-------
(1) Sole-source contracts awarded on the basis of a unique
capacity to perform;
(2) Various set-aside programs, which allow for limited
competition solely among small businesses and/or firms
in labor surplus area; and
(3) Procurements set aside for minority-owned firms under
Section 8(a) of the Small Business Act. (These pro-
curements may be awarded directly to such firms without
competition.)
Set-aside programs are part of the Federal Government's
efforts to promote the economic viability of small busi-
nesses, particularly minority-owned businesses.
The sponsoring office also may procure technical services
through grants, cooperative agreements, and interagency
arrangements.
The major drawback of all other procurement options is
that, while they may serve other Agency goals, they re-
strict competition in one way or another. Some also ex-
clude large survey research organizations and nonprofit
organizations, including academically-based research groups
which often are especially qualified for major Agency
surveys.
D. THE PRINCIPAL PARTICIPANTS
This section describes the roles of four major partici-
pants in a typical EPA-sponsored contract survey, namely --
The project officer;
The contracting officer;
Agency technical advisors such as statistical experts
and information management specialists; and
The contractor.
1. Role of the Project Officer
The "prime mover" of any Agency survey is the person
selected by the Division Director (or Office Director)
of the sponsoring office to plan and manage the pro-
ject. For a contract survey, this person is called
the "project officer" or sometimes "project technical
officer. (if the survey is executed solely by EPA
-5-
-------
staff and no contract is involved, the person direc-
ting the survey is called the "project manager.")
The project officer usually is given the lead respon-
sibility for technical oversight of the survey early
in the planning stage and holds this position until
the contract is closed out. He/she is responsible to
select staff to develop the technical requirements of
the survey contract and make sure the contractor ful-
fills these requirements.
The project officer also carries out various advisory
and administrative tasks in support of the Procurement
and Contracts Management Division (PCMD), which has
overall authority for the contract awards process.
Exhibit 1 lists the project officer's principal respon-
sibilities during each phase of a typical contract sur-
vey and the chapters detailing these responsibilities.
We also have provided a "Help Directory," Exhibit 2,
listing sources to contact for further assistance in
designing the survey, preparing the survey schedule and
budget, initiating the procurement, or obtaining clear-
ance to collect the needed data from the Office of
Management and Budget (OMB), as required by the Paper-
work Reduction Act of 1980.
2. Role of the Contracting Officer
The acquisition of outside technical services to support
the development and implementation of a contract survey
is a complex process governed by laws, regulations,
policies, and procedures defined in Federal and EPA pro-
curement regulations and, the EPA Contracts Management
Manual. A specialized knowledge of these requirements
is especially important in a competitive procurement.
The "contracting officer" is the person delegated by
PCMD to work with the project staff in obtaining the
best possible technical support given the funds avail-
able for the contract. The contracting officer over-
sees all aspects of the procurement process to ensure
that it is carried out in accordance with all perti-
nent rules and regulations.
It is easy to see why it is essential for the project
officer and the contracting officer to maintain close
cooperation during all phases of the survey. While
the project officer has full responsibility for the
technical aspects of the procurement both before and
-6-
I
-------
EXHIBIT 1
THE PROJECT OFFICER'S RESPONSIBILITIES
DURING A TYPICAL CONTRACT SURVEY
CHAPTER
SURVEY PHASE
PRINCIPAL TASKS
2
PREPLANNING
Preparing a needs assessment to
determine whether the survey is
both necessary and feasible.
3
PLANNING AND
DESIGNING
THE SURVEY
Establishing the survey design
requirements.
Developing a preliminary schedule.
Developing a preliminary budget.
4
INITIATING
THE
PROCUREMENT
Initiating the request to procure
outside technical support.
Developing the procurement request
package.
5
WRITING THE
STATEMENT OF
WORK
Defining the scope of work of the
procurement.
6
SELECTING
THE
CONTRACTOR
Safeguarding information from
of ferors.
Evaluating and scoring the tech-
nical proposals.
Supporting the contract negotia-
t ions .
7
MANAGING
THE
CONTRACT
Fulfilling the Agency's contractual
commitments.
Obtaining OMB clearances.
Monitoring and controlling the
contractor's technical performance.
Assisting in the administrative
oversight of the contract.
8
REPORTINC
AND RELEASING
THE FINDINGS
Reviewing and clearing drafts of
the final reports.
Releasing the survey findings.
-7-
-------
"HELP" DIRECTORY
EXHIBIT 2
FOR HELP IN DESIGNING THE SURVEY --
Statistical Policy Branch (SPB), Chemicals and
Statistical Policy Division, Office of Standards
and Regulations.
Your office's statistical survey experts.
Volume II of this handbook.
What Is a Survey? -- a small, easy-to-read booklet
published by the American Statistical Association,
Washington, D.C., available from SPB.
Payne, S.; The Art of Asking Questions; Princeton
University Press, Princeton, N.J., 1951.
Schuman, H., and Presser, S.; Questions and An-
swers in Attitude Surveys; Academic Press, 1982.
Slonim, M.; Sampling in a Nutshell; Simon and
Shuster, 1960.
FOR HELP IN SCHEDULING AND BUDGETING THE SURVEY --
Other project officers.
Your contracting officer.
Your office's statistical staff or SPB.
Informal contacts with current or past EPA survey
sponsors.
FOR HELP WITH EPA PROCUREMENT POLICIES AND PROCEDURES --
Your contracting officer.
The EPA Contracts Management Manual, available
in all Division-level offices.
The EPA Project Officers' Handbook.
FOR HELP IN OBTAINING 0MB CLEARANCES --
The Information Management Branch (1MB),Regulation
and Information Management Division, Office of
Standards and Regulations.
Your office's Information Management Coordinator.
-8-
-------
after the contract award, the contracting officer is
fully responsible for overseeing the business and ad-
ministrative aspects.
3. Role of Other Agency Specialists
At various stages of the survey's development, the
project staff may draw upon the expertise of Agency
statistical and information management experts as well
as outside sources having specialized knowledge about
the subject area of the research.
Most Agency program offices have statistical experts
who can be tapped during the course of the project to
help design the survey, evaluate the proposals submit-
ted by the prospective contractors, etc. In addition,
the Statistical Policy Branch (SPB) of the Office of
Standards and Regulations (OSR) has statistical experts
available on an "as needed" basis. SPB can help the
project staff develop the survey plan or review docu-
ments prepared by the contractor (e.g., the proposal,
the questionnaire, the sampling plan, etc).
Each program office also has an Information Management
Coordinator, who can help you prepare and expedite
the clearance required by the 1980 Paperwork Reduction
Act for all information collections. The Information
Management Branch (1MB) of OSR manages the Agency's
clearance process under the Act. 1MB can help you
obtain the necessary approvals for the survey from
the U.S. Office of Management and Budget and other
reviewing authorities.
4. Role of the Contractor
The contractor, who serves as an agent of EPA during
the term of the award, typically is an experienced
survey research organization in the private sector.
The contractor generally must be capable of providing
a wide range of technical and support services to help
the Agency design and execute all aspects of the sur-
vey. Survey contractors frequently have a permanent
staff of sampling experts, data processors, data ana-
lysts, and a network of trained interviewers.
EPA contracts out much of its survey work to relative-
ly large survey research firms because monitoring and
controlling the data collection and data processing
activities in most Agency surveys requires dozens of
trained interviewers, field coordinators, computer spe-
cialists, and statistical analysts. Few small research
-9-
-------
research firms have the resources needed to quickly
and efficiently gather and process information from the
many hundreds of widely dispersed respondents usually
involved in Agency surveys.
In this chapter we examined the reasons statistical
surveys are important to the Agency's regulatory and
enforcement programs. We also described the major
types of surveys, the contracting options available to
the sponsoring office for obtaining survey support ser-
vices, and the roles of the principal participants in an
Agency-sponsored survey.
The next chapter will show you how to to determine at
the outset of the project whether a survey is necessary
and feasible.
-10-
-------
CHAPTER 2
HOW TO DECIDE WHETHER TO DO THE SURVEY
Before the sponsoring office begins to draft a formal plan for
the survey, a preliminary assessment of the necessity and feas-
ibility of the proposed research should be made. This tasR is
often delegated to the person who will lead the project.
This chapter gives instructions for preparing an informal re-
port (a "needs assessment") to demonstrate whether the proposed
survey --
ฎ Is necessary -- there is a clear need for
certain information that is not available
elsewhere in exactly the form the Agency needs
it; and
m Is feasible -- a survey appears to be the
most cost-effective, least burdensome way to
help fulfill a particular legislative mandate
or program goal.
Much of the information in the assessment report will be helpful
later for preparing the survey plan (see Chapter 3), the paper-
work for initiating the procurement (Chapter 4), and the request
for OMB clearance of the survey (Chapter 7). The amount of de-
tail that should be included is optional. However, since the
purpose of the report is to give officials of the sponsoring
office basic information to make a decision on the need for a
survey, generally, it should not be more than two pages long.
A. IS THE SURVEY NECESSARY?
To demonstrate the necessity of the proposed survey, the
report should affirm the truth of the following two state-
ments :
(1) EPA has a clear need for certain new information.
The information you propose to collect from the public
cannot simply be "nice to have." There has to be a real
need for it -- to carry out a particular statutory,
regulatory, or judicial mandate or program goal, for
example. Preferably, you should be able to cite chap-
ter and verse from the specific statute or court order
-11-
-------
giving the Agency a direct mandate to collect certain
information. Otherwise, your assessment should indi-
cate the legal or judiciary authority requiring the
Agency to take certain actions and the reasons EPA
cannot do so unless the new data are obtained.
(2) The information is not available elsewhere.
Once you have shown why the Agency needs the informa-
tion, you will have to do a data search to demonstrate
that the information (a) is not available from some
existing source (published or unpublished) either with-
in or outside the Federal Government, or (b) could not
be obtained some other way such as by "piggybacking"
on a survey being conducted by another agency.
Your data search may have other benefits. The exper-
iences of the people who have done similar surveys
may help you avoid costly methodological errors. You
may find that the questionnaire used in a previous
survey needs only slight modifications to meet your
needs. You may discover that another agency is cur-
rently collecting similar data that will enable you
to get the information you need more quickly and re-
duce the scope of your information collection effort.
On the the other hand, your data search may reinforce
the necessity of doing a new survey. Even if some of
the information you need is available from other
sources, you may find that --
9 The source won't release it (for confidentiality
reasons, perhaps);
ป It is outdated, not in the form you need it, or is
incomplete;
The methods used to obtain the data are questionable
and, therefore, the statistics may not be reliable
enough for your purposes; or
9 It is not representative of the population you are
most interested in or covers only a limited geo-
graphic area.
IS THE SURVEY FEASIBLE?
Your needs assessment also should clearly demonstrate the
feasiblity or workability of the survey. While you may be
able to justify the necessity of obtaining new data, there
-------
are still some practical issues to consider. The assess-
ment, therefore, must also show that all three of the
following statements are true --
(1) Sponsoring a survey appears to be the best way to ob-
obtain the needed information.
Although the Agency may indeed need certain informa-
tion, it simply may not be available. For example,
most obvious data sources may not keep records that
will provide the information you need.
Or, after consulting with a survey expert, you may
find that collecting the particular data you need would
impose too heavy a burden on the prospective respond-
ents. Since the Paperwork Reduction Act prohibits
Federal agencies from imposing undue burden on the
public in gathering information, you may have to use
other research techniques to obtain the desired data.
(2) There is time to wait for the results.
If the objectives of your survey are very limited, you
may be able to do a "quick and dirty" data collection
effort in just a few weeks using Agency personnel.
But if you expect to hire a contractor to collect data
from several hundred people, you are not likely to get
the results for at least eight months -- and eighteen
months is not unusual. First, a major contract survey
takes months to design and plan. Second, the Federal
procurement process for a competitive contract award
may take five months or more. Finally, obtaining the
required internal reviews and a clearance from the
U.S. Office of Management and Budget (OMB) usually
takes at least 60 days after the contractor is aboard;
the contractor cannot begin to collect data until OMB
approves the survey if ten or more people are to be
interviewed.
(3) You have sufficient funds to get the information you
need.
Surveys can be expensive as well as time-consuming. To
make a reliable survey of, say, 400 firms could easily
cost $100,000. Even a relatively small data collection
project involving 25-50 businesses or industries might
cost $30,000-$50,000. A national survey of 5,000
people using personal interviews could cost as much as
$3 million.
_1 3-
-------
The method and techniques you use to collect the data
also will affect the cost of the survey. Generally
speaking, if personal interviews are used, the survey
will cost at least twice as much as telephone inter-
views and take twice as long. Both methods are more
costly than a mail survey. By clustering the respon-
dents at a few central locations, the costs of doing
personal interviews can be substantially reduced. For
example, personally interviewing the employees of a
particular firm during working hours would cost far
less than surveying them at home by mail or by phone.
Use of a mass-administered questionnaire is another
cost-saving option that may be appropriate for some
surveys.
(See Chapter 2 of Volume II for information on the
advantages of different data collection methods.)
If your assessment shows that all five of the above statements
are true, submit a brief report to your Office Director for ap-
proval. If you expect the project to cost more than $2 million,
it is best to get the approval of your Assistant, Deputy, or
Regional Administrator before proceeding with the project.
In this chapter we showed you how to prepare a needs
assessment, an informal preplanning document designed
to help officials of the sponsoring office decide whe-
ther to commit staff and funds to the development of a
full-scale survey plan.
Once the needs assessment is approved, the project staff
can begin to develop a formal planning document. This
is the subject of the next chapter.
-14-
-------
CHAPTER 3
PLANNING THE SURVEY
This chapter gives instructions for preparing a formal survey
plan. Specifically, it will show you how to --
Define the research objectives and
the prin-
cipal design specifications of the
survey;
Prepare a schedule for the in-house
and con-
tract resources needed to carry out
the
project; and
Prepare a project budget.
A carefully thought-out plan is important to the successful
outcome of any survey. It is essential for a contract survey.
Once the sponsoring office has determined that a new survey is
both necessary and feasible, a responsible official -- usually
the person who later serves as the project officer -- should be
appointed to develop a plan for the survey.
A formal planning document will help the sponsoring office co-
ordinate the survey more effectively with --
= Contracts personnel (to initiate the procurement of outside
technical support);
= Information management specialists (to obtain the internal
and external clearances required to collect data from the
public); and
Survey experts (to develop a sound, workable methodology).
A. ESTABLISHING THE SURVEY DESIGN REQUIREMENTS
The first step in developing a formal survey plan is to
establish a set of design specifications to guide the
research.
-15-
-------
At a minimum, the plan should specify
(1) The research objectives (stated in terms of the kinds
of information to be collected and the problems to be
solved) and the relationship of these objectives to
the Agency's Mission;
(2) The population to be investigated and the extent of
coverage;
(3) That probability sampling must be used to select the
respondents from the target population;
(4) The level of precision (sampling error) the Agency
will allow for one or more of the key statistics; and
(5) A minimally acceptable rate of response for the survey
as a whole.
Of course, you may specify the Agency's requirements for
any number of other design elements such as the data col-
lection and data processing procedures. However, these
additional specifications will not substitute for clear
definitions of the above five design elements.
Below we (1) expand on these five minimum design specifi-
cations and (2) briefly discuss nine other design elements
often covered in survey plans.
1. Minimum Design Specifications
Research objectives.
The first fundamental design specification of any
survey plan is a clear statement of the objectives
of the research. The statement should define (a)
the information to be collected, (b) the problems
to be solved, (c) the hypotheses to be tested and,
if possible, (d) the key study variables.
For example, let us say that, in conjunction with
the development of revised safe drinking water
standards, you have decided to investigate the
effects of a chemical substance suspected of con-
taminating the drinking water of communities where
certain industries are operating. You might state
the overall objective of the survey as follows --
"To examine the health status of the adult popu-
lation residing near industries generating Sub-
stance X."
-16-
-------
You could be more specific and state your objectives
this way --
"To investigate the exposure levels of the popula-
tion residing near industries generating Sub-
stance X as a product or byproduct of certain
manufacturing processes;
"To investigate possible changes in the health
status of the subject population; and
"To investigate the attitudes and opinions of the
local medical community concerning any changes
in the health status of the subject population."
You could be still more specific and include (a) the
particular industries and communities you plan to
investigate, (b) the precise industrial processes
suspected of contaminating the drinking water, and
(c) the reasons Substance X is believed to be caus-
ing the problem.
If at all possible, you should also include one or
more key variables in your statement. Continuing
with the above example, your objectives statement
could indicate your intent to measure the health
status of the subject population by questioning
local residents about (a) "the number of visits they
made to a physician in the past 24 months," (b) "the
number of prescription drugs taken daily," or (c)
"any significant change in the taste of the local
drinking water" or "...in the clarity of the water."
It is also important to relate the survey objectives
to specific regulatory or statutory mandates. Cite
any relevant mandates and briefly explain how the
survey is expected to support them. To illustrate,
the statement of the research objectives for the
above example would include pertinent citations from
the Safe Drinking Water Act and any Agency regula-
tions developed to carry out the Act.
(An exemplary statement of the research objectives
and relevant legislative authority for a recent
Agency survey on carbon monoxide exposure is shown
in Appendix A.)
Target population and coverage.
The second fundamental design specification of any
survey plan is the "target population" (population
of interest) and the "coverage" of the survey.
-17-
-------
The target population is simply all the people,
businesses, or other population units chosen for
investigation. The target population does not have
to be individuals. It can be households, business-
es, institutions, or any other population unit.
Depending on your research objectives, the target
population could be "all the people in the entire
Nation," "all adults 18 years and older," or it
could be much more specific -- "drivers who commute
long distances by automobile" or "people over 65
living in cities with high pollution levels." An
appropriate target population for our hypothetical
survey might be "the adult population of communities
where industries suspected of generating Substance X
are located."
The coverage of the survey may be a particular geo-
graphic area (e.g., "the Northeast Corridor," "the
entire Nation") or a subgroup of the target popula-
tion (e.g., "high risk individuals 65 or older,"
or "chief engineers of hazardous waste disposal
plants").
While defining the target population and the cover-
age appears to be a simple matter, it often is very
difficult. For example, you may know what you want
to study (Substance X, say), but you may not know
where to find an appropriate group of people to
study or which subpopulation in that group is at
greatest risk.
(See Appendix A for a description of the target
population used in the above mentioned EPA survey.)
Use of probability sampling.
The third fundamental survey design specification
is a requirement that probability sampling must be
used throughout the selection process to identify
the survey respondents. The "respondents" are the
specific individuals in the target population which
the researchers contact in person, by phone or by
mail to get the information the Agency needs.
Sampling is a method of selecting a representative
segment of the target population to obtain precise
estimates of the opinions, attitudes, beliefs of
behavior of the larger population from which the
sample is drawn (the target population). Most
acceptable sampling methods, including probability
-18-
-------
sampling, are well grounded in statistical and
probability theory.
Using "probability sampling" means that every unit
(person, household, etc.) selected to represent the
target population has a known, nonzero chance of
being included in the sample. This method of selec-
ting the survey respondents makes it possible for
the researchers to draw statistically valid infer-
ences about the entire population the sample is
designed to represent.
Today, virtually all surveys that can be considered
reliable or representative use some method of prob-
ability sampling. Conducting a "census" of the en-
tire population of interest is both impractical and
inefficient. Even when the target population is
relatively small, it is usually more cost-effective
to obtain information from a sample rather than in-
terview everyone in the population.
It is not necessary at the planning stage to actu-
ally select the sample, determine what size it
should be, or specify anything else about it. That
can be done later by Agency sampling experts or,
preferably, by the contractor. The only important
thing at this point is to incorporate a requirement
in the plan that probability sampling must be used
throughout the selection process.
(See Chapter 4 of Volume II for more information on
sampling.)
Required level of precision.
The fourth fundamental design specification deals
with the level of precision the Agency will require
for one or more of the key statistics. (The "sta-
tistics" are the results obtained from tabulating
data collected from a sample.)
The level of precision you specify will depend on
a great many factors -- the purposes of the survey,
the intended use of the data, the time you have
available to collect the data, and particularly how
much you have to spend. Whatever level you specify
should be stated in terms of sampling error, which
is the difference between the statistics and the
values you would have obtained had all the members
of the target population been surveyed.
-19-
-------
Again using our drinking water example, suppose that
70 percent of the sample detected significant chan-
ges in their water quality. Undoubtedly a survey
of the entire target population would have produced
a somewhat different percentage. The advantage of
using probability sampling is that the rules of sta-
tistics allow you to estimate the difference between
the percentage reported by the sample and that which
would have resulted from a complete enumeration of
the target population. The margin of error that re-
sults when sampling is used is measurable.
The margin of error of the above statistic (70 per-
cent) should be stated in terras of a number of per-
centage points above or below the stated value of
the percentage -- + 3 percent, say. This would in-
dicate that between 67 and 73 percent of the "true"
population would have detected a significant change
in their drinking water (had they all been surveyed).
On the other hand, to show the margin of error of a
mean (instead of a percentage), you report the num-
ber of units above or below the calculated average
value. For example, if the average age of the sample
who responded to this particular drinking water
question was 34 years, you would report the sampling
error as + a certain of years.
We turn now to nonsampling errors -- errors that are
not attributable to sampling methodology. Unlike
sampling errors , nonsampling errors are not easily
measured.
Some nonsampling errors occur at random and tend to
have virtually no effect on the validity of the
statistics. Examples are response errors occurring
as a result of an occasional deliberate or uninten-
tional reply, such as deliberately overstating or
understating personal income.
There is another kind of nonsampling error, however,
that tends to create errors in the same direction
over the entire sample. These systematic, one-sided
errors are called "biases." Biases are undesirable
because they can distort the results in ways that
are difficult to measure.
Biases can arise at any point in a survey due to
methodological or management problems, such as fail-
ure to interview all the people who were supposed
to be interviewed, faulty sampling frames (sources
-20-
-------
from which the sample is drawn) , po
the questionnaires, or inexperienced
or wording of
interviewers.
Target response rate.
The fifth and last fundamental design specification
concerns the desired overall response rate from the
sample -- the "target response rate."
Establishing a target response rate, like establish-
ing a certain level of precision, is a way of build-
ing quality into the survey. Unless a large propor-
tion of the sample provides the information EPA is
seeking, the survey results will have little validi-
ty. The reason is that, no matter how carefully the
sample is drawn, there is no valid way of inferring
the opinions, attitudes, or beliefs of the members
who do not respond. A high rate of response also
helps to prevent bias.
When you specify the target response rate, be sure
to define what you mean by "response rate." Usually
it is defined as the ratio of --
Number of completed interviews (or questionnaires)
Total number of eligible units in the sample
As for determining the rate itself, that is some-
thing a survey expert will have to help you with.
There is no ideal rate that can be applied to all
surveys or even to different types of surveys.
Generally speaking, a 75 percent response rate is
acceptable for an in-person or telephone survey.
Mail surveys of the general public usually yield
much lower response rates -- less than 40 percent
is not uncommon -- even after several repeat mail-
ings. However, telephone calls or personal visits
to a subsample of those who have not returned their
questionnaires can raise the initial response rate
rate to a mail survey considerably. In most cases,
unless an extensive follow-up of the nonrespondents
is planned, a mail survey where responses are vol-
untary is an unacceptable method of collecting data
for Agency-sponsored surveys.
An adequate follow-up of nonrespondents is essen-
tial for achieving a satisfactory response rate in
almost all surveys, whatever method is used to col-
lect the data. Follow-up techniques include return-
ing to households where no one was home and making
-21-
-------
a second (or third) attempt to persuade people to
answer questions after they refused when first
contacted. (See Chapter 5 of Volume II for more
information on nonresponse.)
To recap: The above five design specifications are es-
sential to ensure the quality of the data and the sta-
tistical findings. Of course, your survey plan need
not be limited to these key specifications. Depending
on the statistical expertise available to the project
staff, any number of design specifications can be
defined by the Agency. However, none will contribute
as much to the successful outcome of the survey as
the above five specifications.
2. Optional Specifications
The nine optional design elements discussed below are
often covered in survey plans: (1) pretests, (2) data
collection methods, (3) sampling frames, (4) the sam-
pling plan, (5) questionnaires, (6) data collection
procedures, (7) data processing procedures, (8) the
analysis plan, and (9) reports and other deliverables.
For most of these elements, we recommend that you wait
and let the contractor develop specifications for them.
(1) Pretests.
Pretests are mandatory for all surveys. For most
surveys, a pilot test -- an even more elaborate
pretesting procedure -- is highly advisable. In
a pilot test both the field procedures and the
questionnaire are tested in advance, usually on a
larger sample than in a pretest.
Either you or the contractor can pretest the work-
ability of the questionnaire or the methodology
on a small sample.. If you intend to conduct the
pretest, explain how you plan to go about it.
(See Chapter 3 of Volume II for more information
on pretests.)
(2) Data collection methods.
It is not necessary to specify a particular data
collection method in the plan -- mail, or face-to-
face or telephone interviews, or some combination
of methods -- unless you have a special reason
-22-
-------
for doing so (e.g., a successful experience with
a particular method under similar conditions) .
Except in unusual circumstances, in fact, we
recommend that you leave the choice up to the
contractor. Specifying the data collection meth-
od tends to impose unnecessary restrictions on
whoever does the final survey design.
(See Chapter 2 of Volume II for more information
on choosing data collection methods.)
Sampling frames.
A "sampling frame" is the source (or sources) from
which the sample is drawn. Often it consists of
one or more lists of individuals or organizations,
but it also may be a set of city blocks, a set of
telephone exchanges, etc.
Sometimes the Agency will supply the sampling
frame (e.g., a list of hazardous waste disposal
firms). For other surveys, you may leave the con-
struction of the frame entirely up to the contrac-
tor. If you are going to furnish the frame, how-
ever, indicate this in your survey plan. Also
note when it will be available.
Sampling plan.
A sampling plan describes and justifies the choice
of the (a) sampling units, (b) sample size(s), and
(c) methods of selecting the sample from the
frame. The sampling plan is a vital component of
the final survey design, but it is not necessary
for the Agency to develop it. Usually that is
best left to the contractor. (You do have to de-
fine the target population, however; without it,
the contractor cannot develop the sampling plan.)
It is especially important not to specify the size
of the sample either now or in the Request for
Proposals (RFP). Specifying both the required
level of precision (sampling error) and the sample
size you desire imposes too many constraints on
the contractor.
In all matters concerning sampling we strongly
advise that you consult a sampling expert. Sam-
pling is I complex "science. A survey sample
may range from a simple random sample to a highly
complex set of samples involving multiple stages
-23-
-------
(or levels of selection) with stratification and/
or clustering of the sampling units into various
groupings. The firm the Agency ultimately hires
to do the survey design will undoubtedly have a
sampling expert develop the sampling plan. (A
sampling expert is a specialist in a certain
branch of statistics. Not all statisticians have
expertise in this area.) We also recommend that
an Agency sampling expert review the contractor's
sampling plan before the data collection begins.
(See Chapter 4 of Volume II for information on
all aspects of sampling.)
(5) Questionnaire(s).
A well-thought-out questionnaire is one of the
most important elements of a successful survey.
A poorly designed questionnaire is likely to pro-
duce serious biases and can significantly affect
the quality of the results.
Both the content and form of the questionnaire
are critical to the outcome of any survey. Ques-
tions should be clearly and unambiguously phrased,
easy to read, and arranged in logical order with
related items grouped together. The form should
be easy to complete and as brief as possible. It
should contain clear instructions on whether cer-
tain questions may be skipped (particularly if the
questionnaire is to be self-administered), where
and how to return the questionnaire, etc.
(See Appendix B for an example of a questionnaire
used in a recent Agency survey.)
While we do not recommend that you complete the
questionnaire at this point in the survey's de-
velopment -- that may be left entirely to the con-
tractor, you should draw up a draft questionnaire
or at least a suggested list of questions for the
survey plan. This also will help you define the
research objectives and estimate the survey costs.
(See Chapter 3 of Volume II for complete guide-
lines on preparing a survey questionnaire.)
(6) Data collection procedures.
Generally you will not need to include specifica-
tions for the data collection (or interviewing)
-24-
-------
procedures in your survey plan. However, when
you prepare the statement of work for the procure-
ment request, you may want to specify certain as-
pects of the collection methodology (data rights,
confidentiality provisions, limits on the length
of the interviews, etc.). These procedural as-
pects are discussed in Chapter 5. (Also see Chap-
ter 5 of Volume II for a detailed discussion of
interviewing.)
Data processing procedures.
It is also unnecessary to specify the data pro-
cessing procedures in the survey plan. Chapter 5
gives instructions for stipulating certain pro-
cessing procedures in your procurement request
such as the format of the data tabulations.
(See Chapter 6 of Volume
on data processing.)
Analysis plan.
II for more information
Although we do not recommend that you specify
either the data collection or data processing
procedures, it is extremely important to draw up
a broad outline of what you expect to get from
the survey. In much the same way as you trans-
lated the research objectives into a preliminary
questionnaire, you should draw up a preliminary
analysis plan showing the purposes and objectives
of the survey -- how responses to key questions,
when tallied, will achieve those objectives.
Your analysis plan also should include of a set
of dummy tables ("tables shells") for reporting
the results of the data tabulations. The dummy
tables, in turn, will help you frame your research
objectives and draft the questionnaire.
(See Chapter 1 of Volume II for details on pre-
paring an analysis plan.)
Reports and other deliverables.
It is not necessary to list all the survey deliv-
erables in your plan. However, we recommend that
you specify and schedule any written reports
(e.g., draft, final interim, special, etc.) as
well as progress reports that will have to be
prepared. You will need this information to
-25-
-------
develop a preliminary schedule for the project,
discussed in B below,
(For more information on deliverables, see Chap-
ter 4, particularly Exhibit 4. Also, Chapter 8
presents suggestions for the form and content of
the final survey report.)
B. SCHEDULING CONSIDERATIONS
This section explains (1) how to prepare a project sched-
ule, (2) the two principal time constraints on a contract
survey, and (3) how to phase the survey.
1. Preparing the Schedule
The second component of the survey plan is a schedule
for the project. You should at least schedule (a) the
project milestones, and (b) the major activities the
contractor and the project staff will have to complete
to achieve each milestone, including the delivery
dates for all deliverables.
To prepare the schedule, first identify the project
milestones and plot them on graph paper. Typical
milestones for a contract survey are --
(1) Submission of the paperwork to hire outside tech-
nical support (the "procurement request package");
(2) Award of the contract;
(3) Completion of the sampling plan, questionnaire,
and analysis plan;
(4) Approval of the clearance package by the U.S.
Office of Management and Budget (OMB);
(5) Completion of the pretest;
(6) Completion of the data collection operations;
(7) Completion of the data processing and analysis;
(8) Completion of the draft of the final report;
(9) Completion of the final report; and
(10) Release of the survey findings.
-26-
-------
Next, list the major activities (and the individual
tasks, if possible) the contractor and the project
staff will be responsible for, and indicate the number
of work hours required for each activity. Then plot
the duration of each activity on your graph.
Your list of contractor activities might include --
(1) Completing the sample design;
(2) Completing the questionnaire and analysis plan;
(3) Pretesting the questionnaire and the data col-
lection methodology;
(4) Evaluating the pretest results and revising the
questionnaire and the methodology accordingly;
(5) Hiring and training the interviewers;
(6) Implementing the data collection procedures (con-
ducting interviews or mailing out the question-
naires) ;
(7) Performing follow-ups, quality control, and data
verif ications;
(8) Editing and coding the data;
(9) Tabulating and analyzing the data; and
(10) Drafting the final report.
Of course, there are some important activities the
contractor cannot perform. Your schedule also should
include the activities(and tasks) the project staff is
responsible for, along with estimates of the number of
work hours neecfed to complete them.
Project staff activities generally include --
(1) Preparing the statement of work and other parts
of the procurement request package;
(2) Supporting the Procurement and Contracts Manage-
ment Division (PCMD) in selecting a qualified
contractor (evaluating the technical proposals
submitted in response to the RFP, participating
in contract negotiations, and performing other
administrative tasks involved in getting the con-
tractor aboard);
-27-
-------
(3) Obtaining OMB clearance for the pretest and the
survey proper;
(4) / Monitoring and controlling the contractor's prog-
ress ;
(5) Reviewing all drafts of contract reports;
(6) Issuing the survey findings; and
(7) Assisting in closing out the contract.
Although it may be difficult to estimate the probable
duration of some project staff activities because much
depends on people outside the project (OMB, the con-
tracting officer, other internal review authorities),
your schedule should include rough estimates for all
major activities. If you need help, contact the Sta-
tistical Policy Branch (SPB) of OSR or a statistician
in your office.
2. Major Administrative Time Constraints
The principal administrative time constraints on any
contract survey are (a) getting the contractor aboard
and (b) obtaining OMB clearances of the pretest and
the actual survey.
Procurement of the contractor.
The project schedule should include the time neces-
sary to get the contractor aboard. Your contracting
officer can help you estimate this but, generally
speaking, much depends on the dollar amount of the
proposed procurement and whether you plan to re-
quest a competitive or a sole-source acquisition.
First, you must allow adequate leadtirae for approval
of the procurement paperwork by your own office. If
any special resources will be required (ADP equip-
ment or services, tape recorders, mailing lists,
etc.), other Agency offices have to approve the
procurement package before it goes to PCMD. (See
Chapter 4 for more information on procurement re-
quirements .)
Finally, the Procurment and Contracts Management
Division must process the procurement. The standard
leadtime for PCMD approval of small purchases (up to
525,000) ranges from 7-80 calendar days, depending
on the size of the award and whether it is to be a
-28-
-------
sole-source or competitive order. New sole-source
or competitive awards over that amount regularly
require 120-240 days for PCMD approval. (See Appen-
dix D for more specific estimates of the leadtimes
required for various types of contracts.)
Obtaining OMB clearances.
To meet requirements of the 1980 Paperwork Reduction
Act, EPA is required to obtain OMB approval of any
information activity involving ten or more respon-
dents. Therefore (except in dire emergencies),
neither Agency staff nor the contractor are permit-
ted to begin collecting data for the survey -- or
even the pretest if more than nine members of the
public are to be questioned -- until OMB approves
the survey methodology and the proposed question-
naire.
Obtaining OMB clearance of any data collection ac-
tivity is a lengthy procedure. At EPA the clearance
procedure is managed by the Information Management
Branch (1MB) of OSR.
As a rule of thumb, allow at least two weeks for
each office that must handle and review the clear-
ance package before it reaches OMB"!! The Act allows
60 days for OMB's review (90 days OMB grants itself
an ex tens ion) .
The project officer is generally delegated the re-
sponsibility (a) for preparing the necessary paper-
work to obtain the OMB clearance, (b) for submitting
the paperwork through proper channels, and (c) for
answering any questions the reviewers may raise
during the clearance process.
If you have developed a comprehensive survey design
and a preliminary questionnaire, you can initiate
the OMB clearance request before the contractor is
hired. However, in actual practice, clearance re-
quests for surveys (or even pretests) are rarely
initiated until after the contractor is aboard.
This is because the contractor usually plays a
major role in designing the questionnaire and the
data collection procedures.
To prevent any unnecessary delays in clearing the
survey, contact OSR's Information Management Branch
early in the planning stage -- preferably as soon
-29-
-------
as you have tentatively defined the objectives of
your survey objectives. IBM can advise you on the
Agency's clearance requirements and give you a more
precise estimate on how long internal and external
approvals of your particular survey will take.
Later chapters of this handbook provide additional in-
formation that may help you estimate the time needed
to obtain OMB clearances (Chapter 7), hire a contrac-
tor (Chapters A and 6), monitor and control contractor
performance (Chapter 7), and review and disseminate
the findings (Chapter 8).
3 . Phasing the Survey
If you have any serious doubts about the Agency's abil-
ity to achieve the research objectives, it may be wise
to do the survey in phases. Two appropriate ways are --
To issue an RFP solely for the pretest (or pilot
test), with the stated intention of issuing another
RFP for the survey if the trial phase indicates the
proposed methodology is workable; or
To specify a series of optional work segments in
the RFP, to be authorized by the Agency when and if
certain concerns are resolved.
The first alternative is obviously more time-consuming
because two RFPs are involved and two 0MB clearances
will be necessary if you plan to use more than nine
respondents in the pretest.
If you decide to use a phased approach, discuss your
intentions with your contracting officer as soon as
possible. PCMD can advise you of other phasing options
that may be available in your particular case.
C. BUDGETING CONSIDERATIONS
The third component of the survey plan is the project
budget. A preliminary cost analysis of the in-house and
contract resources the Agency will need to carry out the
survey is important because the amount of money available
for the project can profoundly affect the design.
If you have prepared your schedule along the lines sug-
gested in the previous section, developing a budget is
relatively easy. Briefly put, you will have to compute
cost estimates for each item on the schedule, separately
-30-
-------
estimating in-house and contractor resources. Be sure to
include costs for computer time, travel, postage, repro-
duction, and printing in addition to estimates for project
staff and contractor personnel.
If you need help in making your estimates, contact SPB, a
statistical expert in your office, your contracting offi-
cer, or other project officers.
When you complete the first draft of the budget, you may
find it necessary to modify the original survey design if
the funds available for the project are insufficient to
meet your design criteria. If so, consider reducing the
coverage, the number of questions, or the target response
rate. If you cannot modify the design and arrive at a
tolerable margin of error given the available funds, you
may have to abandon the idea of a survey altogether.
To recap: At this point in the survey you should have --
(1) Established a clear need for the survey -- in other words,
determined that certain data are not available elsewhere
and that a survey is the most appropriate way to obtain
the information you need;
(2) Prepared a survey plan that includes criteria for at least
the five critical technical design elements discussed in
the previous chapter;
(3) Given full consideration to the availability and cost of
using in-house resources to do the survey, and determined
that certain technical services should be procured from
outside the Agency;
(4) Estimated the internal and contract resources that will
be required to carry out the planned activities, and deter-
mined that adequate funds and time are available; and
(5) Obtained approval from appropriate officials in your
office to initiate the procurement.
-31-
-------
This chapter gave instructions for preparing a plan
to help ensure the successful outcome of the survey.
We recommended that the Agency's requirements for
certain key design elements be clearly specified and
that both a preliminary schedule and budget be included.
Once the plan is approved, the sponsoring office can
begin to develop the paperwork for procuring whatever
outside technical support is required. That is the
subject of the next chapter.
-32-
-------
CHAPTER 4
PROCURING OUTSIDE TECHNICAL ASSISTANCE
Procuring outside technical support for a statistical survey is
a lengthy, complex process. This chapter discusses the first
phase of the procurement process -- the solicitation phase. It
begins when the sponsoring office submits a formal procurement
request package to the Agency's . Procurement and Contracts
Management Division (PCMD) to initiate a contract award. In
the case of a competitive procurement, the desired end-product
of this phase is a sound, realistic Request for Proposals (RFP)
describing the Agency's technical requirements and conveying
all the information prospective offerors need to prepare a
responsive proposal. Requirements for noncompetitive procure-
ments are similar but less complex.
The next two phases of the procurement process, the evaluation
of the offerors' proposals and the management of the contract,
are discussed in Chapters 6 and 7.respectively.
This chapter examines --
The responsibilities of the project officer
and the contracting officer in preparing a
procurement request or order; and
The paperwork required to support and justify
the procurement of technical support services
for a statistical survey, with special emphasis
on competitive procurements.
A. INITIATING THE PROCUREMENT -- A TEAM EFFORT
The project officer's technical expertise and the
ting officer's knowledge of Federal procurement
ments are both essential for developing a sound
award.
The project officer is usually delegated responsibility for
(a) drafting the scope of work statement for the survey,
(b) preparing a formal procurement request package specify-
and justifying the sponsoring officer's technical needs,
and (c) obtaining the required internal approvals for the
procurement before submitting the package to PCMD.
contrac-
require-
contract
-33-
-------
The procurement package consists of a Procurement Request/
Order (EPA Form 1900-8), a Procurement Request Rationale
Checklist (EPA Form 1900), and other supporting materials.
It specifies the scope of the proposed contract work as
well as the deliverables and reports the contractor is
expected to produce. For a competitive procurement, the
package also must include the evaluation factors that will
be used to weigh the offerors' technical proposals, and
other information that will permit the contracting officer
to write a clear, realistic RFP.
The contracting officer is responsible for (a) processing
the procurement request package and (b) determining what
procurement arrangement will be used to purchase the re-
quired services. In addition, for a competitive procure-
ment, the contracting officer must (c) prepare a Request
for Proposals and (d) make qualified firms aware of the
offering. The contracting officer will draft the RFP
based on the information supplied by the initiating in the
procurement request package.
Ideally, an RFP for a survey --
= Requests the development of or specifies a technically
sound survey design that will meet the Agency's infor-
mation needs, and is realistic, considering the time
and funds available for the project and the capabili-
ties and resources of potential contractors;
Conveys all the information offerors will need to pre-
pare a responsive proposal, including all the terras,
conditions, and provisions the Agency intends to in-
corporate in the contract;
= Asks for all the information that procurement and tech-
nical personnel will need to conduct an accurate and
equitable evaluation of the proposals received; and
= Attracts responsive proposals from enough qualified
offerors to provide full and open competition and
permit a suitable selection.
An RFP specifically includes (a) instructions to offerors
on how to prepare a proposal and submit it to EPA; (b)
instructions for their technical proposal (the kinds of
information offerors are expected to discuss in their tech-
nical proposals); (c) business management instructions (the
date and location of the offerors' conference, if any, the
closing date for the proposals, and how to submit cost
and pricing data); (d) all terms, conditions, and provi-
sions the Agency intends to incorporate in the contract;
-34-
-------
and (e) information on how the offerors' proposals will be
evaluated.
The contracting officer also will insert an announcement of
the procurement in the Commerce Business Daily summarizing
the Agency's procurement requirements and advising prospec-
tive offerors where to obtain a copy of the full RFP.
Obviously the quality of the response to the RFP depend
largely on how well the sponsoring office defines the
Agency's technical requirements in the procurement request.
For this reason, we recommend that you discuss your re-
quirements with your contracting officer as soon as you
have tentatively defined them. Your contracting officer
can help you draft the procurement request package, advise
you on the type of procurement that will best meet your
needs, estimate the length of time it will take to get a
qualified contractor aboard, and generally facilitate the
solicitation phase of the procurement.
B. PREPARING THE PROCUREMENT REQUEST PACKAGE
Below are instructions for preparing a procurement request
package for a statistical survey. The instructions incor-
porate the February 1984 revisions to Chapter 2 of EPA's
Contracts Management Manual.
To initiate a procurement estimated at more than $10,000,
the sponsoring office must submit a package containing
the following documents -- unless otherwise instructed by
the contracting officer --
= A one-page "Procurement Request/Order" (EPA 1900-8),
shown in Exhibit 3 on the next page;
A "Procurement Request Rationale Checklist," shown in
Appendix E, which asks for information on 24 specific
items. The "Checklist" replaces the 14-point document
("Procurement Request Rationale") formerly used to
describe and justify the procurement requirements of
the office initiating the request.
= All applicable supporting documents listed in item 2
of the Checklist, such as a procurement abstract, a
statement of work, technical proposal instructions,
and the competitive technical evaluation criteria.
If the required survey support services are budgeted at
$10,000 or less, a "Procurement Request/Order" ana a one-
page description of the scope of work usually will suffice.
-35-
-------
EXHIBIT 3
(Shaded areas are for use of procurmenl office onhf)
Page
of
US ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, DC 20460
PROCUREMENT
REQUEST/ORDER
1 NAME OF ORIGINATOR
3. MAIL CODE
4. TELEPHONE NUMBER
2 DATE OF REQUISITION
5. DATE ITEM REQUIRED
6. SIGNATURE OF ORIGINATOR
7. RECOMMENDED PROCUREMENT METHOD
~ Competitive Z3 Noncompetitive
8. DELIVER TO (Project Manager)
9 ADDRESS
10. MAIL CODE 11. TELEPHONE NUMBER
12 FINANCIAL DATA
(a) APPROPRIATION
NOTE: ITEM 12 (c) DOCUMENT TYPECONTRACT = "C," PURCHASE
ORDER = "P," IGA - "A," OTHER (Misc.! = "X '
.. FMO USE .
(b) (13 digits!
DOCUMENT
CONTROL NUMBER
(d) (S digits!
ACCOUNT NUMBER
(el (10 digits)
OBJECT
CLASS
jf) (4 digits)
AMOUNT Ig)
DOLLARS
CTS
13 SUGGESTED SOURCE (Name. Address. ZIP Code, Phone/Contact,
14. AMOUNT OF MONEY
COMMITTED IS;
~ ORIGINAL
C INCREASE
~ DECREASE
15. CONTRACTING OFFICE C IS ~ IS NOT AUTHOR-
IZED TO EXCEED AMOUNT SHOWN BY 10%
16. SERVICING FINANCE OFFICE NUMBER
17. APPROVALS
8 BRANCH/OFFICE
DATE
d. PROPERTY MANAGEMENT OFFICER/DESIGNEE
DATE
b. DIVISION/OFFICE
DATE
e. OTHER (Specify)
DATE
c. FUNDS LISTED ABOVE ARE AVAILABLE AND
RESERVED
DATE
e. OTHER (Specify)
DATE
IB. DATE OF ORDER
19. ORDER NUMBER
20. CONTRACT NUMBER (if any) 21 DISCOUNT TERMS
22. FOB POINT
23. DELIVERY TO FOB POINT BY on
24. PERSON TAKING ORDER/QUOTE AND PHONE NO
26. CONTRACTOR (Name, address. ZIP Code)
26. TYPE OF ORDER
~ a. PURCHASE
REFERENCE YOUR QUOTE (See Block 24)
PLEASE FURNISH THE ABOVE ON THE TERMS SPECIFIED ON BOTH SIDES
OF THIS OROER AND ON THE ATTACHED SHEETS. IF ANY. INCLUDING
DELIVERY AS INDICATED. THE PURCHASE IS NEGOTIATED UNDER
AUTHORITY OF 41 USC 262(c) ( ). '
~ b. DELIVERY PROVISIONS ON THE REVERSE ARE DELETED. THE
DELIVERY ORDER IS SUBJECT TO THE TERMS AND CONDITIONS
OF THE CONTRACT. (See Block 20)
~
ORAL
~
WRfTTEN
~
CONFIRMING
27. SCHEDULE
ITEM
DUMBER
la)
SUPPLIES OH SERVICES
(M
QUANTITY
ORDERED
(c)
UNIT
Id)
ESTIMATED
UNIT PRICE
le)
UNIT
PRICE
AMOUNT
M
QUANTITY
ACCEPTED
IM
TOTAL
28. UNITED STATES OF AMERICA
8Y (Sfgnafum)
20. TYPED NAME AND TITLE OF CONTRACTING OFFICER
EPA Form 1900-8 (Rev 4-84) Replaces previous editions.
end EPA Form 1900-8T. which ere obsolete.
-------
The sponsoring office must then obtain all required appro-
vals (shown in Appendix F) before forwarding the procure-
ment request package to the contracting officer for action.
The level of management approvals required by PCMD depends
on the amount budgeted for the support services and the
types of services being requested.
The remainder of this section provides some tips on prepar-
ing (1) your response to some of the principal items on
the "Procurement Request Rationale Checklist" and (2) the
major supporting documents.
1. Checklist Requirements
Of the 24 "items" on the "Procurement Request Rationale
Checklist," 15 are self-explanatory and require only
a "check" in the proper box or a one-line descriptive
comment. The information requested in Items 8, 10, 11,
12, 13 (a and b), 14, 15 and 18, however, is somewhat
more complex. The following will help you complete
them.
Item 8 (Availability of data from other sources).
In Chapter 2 we discussed the importance of prepar-
ing a needs assessment at the outset of the project
to demonstrate the necessity and feasibility of the
proposed information collection activity. Being
able to show that there is a clear need for new
data is basic to obtaining an 0MB clearance for the
survey. It is also of vital importance in obtaining
approval of your procurement request. Refer to this
chapter in preparing this item.
Item 10 (Recommended sources).
This item gives you an opportunity to recommend par-
ticular sources for the procurement. Among the me-
chanisms available to help you compile a list of
sources are --
-- A "Sources Sought" announcement in the Commerce
Business Daily. Ask your contracting officer to
place an announcement of this type in the CBD to
solicit prospective sources for the survey sup-
port services you require. The purpose of this
announcement is to give new contractors in par-
ticular an opportunity to submit a capability
statement and other materials for consideration
by the intiating office and PCMD.
-37-
-------
-- The Procurement Automated Source System (PASS)
established by the Small Business Administration
to increase contract opportunities for small bus-
inesses. Consider selecting prospective offer-
ors from this system, particularly for small
surveys.
-- The directory of professional minority scienti-
fic and technical firms. This current, nation-
wide directory is available from EPA's Office
of Small and Disadvantaged Business Utilization.
It is designed to help procurement and project
officers identify minority firms as prospective
sources for EPA contract work.
Your contracting officer and Small Business repre-
sentative in PCMD can give you more information
on these and other mechanisms designed to promote
competition for Federal contracts.
11em 11 (Anticipated procurement arrangements).
If you expect the survey to cost over $10,000 and
hope to use a noncompetitive procurement, you must
support your request with a Justification for Non-
Competitive Procurement" (JNCP). The two principal
justifications for a JNCP are (a) urgency -- the
amount of time to place the contract is too short to
allow for unlimited competition and (b) uniqueness.
The JNCP should clearly explain (a) the circumstan-
ces that make a competitive negotiation impractical
and (b) the unique or predominant capability of the
prospective contractor. It also should indicate
what consideration has been given to other sources
and the reasons they lack the capability of the
firm under consideration.
Federal Procurement Regulations require that a mar-
ket search or a sources sought synopsis be conduct-
ed for certain noncompetitive procurements.
If you propose to use a noncompetitive procurement
arrangement, discuss it with your contracting offi-
cer before submitting the JNCP.
Item 12 (Appropriateness of set-aside procurements).
See our discussion of set-aside procurement options
in section C of Chapter 1.
-38-
-------
Item 13a (Project schedule).
See our instructions for preparing a project sched-
ule in section B of Chapter 3.
Item 13b (Schedule of deliverables, excluding re-
ports).
Exhibit 4 on the next page lists the typical deli-
verables for a survey project. State a due date
for each deliverable in terras of "work days" or
"calendar days" after the contract award. Use the
project schedule to establish tentative deadlines.
Also indicate the number of copies of each deliver-
able you require.
Item 14 (Options).
See section B-3 of Chapter 3 for a discussion of
phasing options.
Item 15 (Reports).
See our discussion of scheduling in Chapter 3.
Also, section C of Chapter 7 suggests the format and
content of progress reports, and Chapter 8 is en-
tirely devoted to the final report. Be sure to
state that the Agency has review authority over all
draft and final reports and estimate the length of
time required for these reviews.
11em 18 (Procurement budget).
Indicate (a) the funds available for the contract
and (b) the anticipated level of effort (in terms
of person years or similar proxies for dollar
estimates). See section C of Chapter 3 for informa-
tion on budgeting the procurement.
Supporting Documents
Below are a few comments on three of the supporting
documents generally required for competive survey pro-
curements: (1) the scope of work (statement of work),
(2) concise technical proposal instructions, and (3)
competitive evaluation criteria. (The lone supporting
document required for noncompetitive procurements was
briefly described under Item 11 above.)
(For additional help in preparing thee and other sup-
porting documents listed on the Checklist, see PCMD's
-39-
-------
EXHIBIT 4
LIST OF TYPICAL SURVEY DELIVERABLES
1.
Questionnaire(s).
2.
Sampling plan.
3.
Data analysis plan.
4.
OMB clearance justification.
5.
A specified number of blank questionnaires, forms,
instruction manuals, training materials, manual pro-
cessing specifications, and other written materials
to be used in the data collection activities.
6.
Specifications and instructions for the data process-
ing activities, including data cleaning procedures
and specifications, tape record layouts, coding in-
structions manual, and codes.
7.
A full account of the data collection results for
the original sample showing (a) cases determined to
be eligible (complete and incomplete interviews, by
reason), and (b) cases determined to be ineligible,
by reason.
8.
Results of the interview validations.
9.
Information on error rates found during verification
of the coding and keying operations in the data pro-
cessing phase.
10.
Edited data tapes. (Specify any identifiers the
Agency will need to make the survey data compatible
with information in other EPA data banks.)
11.
Tabulations. (Specify the form and content.)
12.
Estimates of sampling error.
-40-
-------
Project Officer's Handbook. Their instructions apply
to all types of procurements, not to surveys in par-
ticular. )
(1) Statement of work.
This is the most critical part of the procurement
request package from the standpoint of the sponsor-
ing office. For this reason, all of Chapter 5 is
devoted to a discussion of the topics that should
be covered in the statement of work for a contract
survey.
The statement of work may vary from a simple des-
cription of the research objectives to a comprehen-
sive plan for the entire survey. Regardless of how
simple or complex,it should clearly define the work
to be performed and the contractor's obligations.
In a competitive procurement, the statement of work
supplies the bulk of the technical information for
the RFP. In a noncompetitive arrangement it out-
lines the scope of work of the project. The state-
ment of work can strongly influence the quality of
the offerors' proposals. Later, possibly with some
modifications during the negotiations, it shapes
the contract.
At a minimum, to permit the offerors to prepare
a technically sound and responsive proposal, the
statement of work should include a clear, unambig-
uous set of survey design specifications covering
at least the five key design elements discussed
in the previous chapter. To reiterate, these five
elements are --
A clear statement of the research objectives
in terms of the kinds of information to be
collected, the problems to be solved, and the
hypotheses to be tested.
A clear and complete definition of the target
population, including a specification of the
reporting units (e.g., individuals, households,
businesses), and the coverage.
A requirement that probability sampling must be
used at all stages of respondent selection and
that EPA has the right to review the selection
procedures.
-41-
-------
The level of precision (sampling error) the
Agency will require for one or more of the key
statistics. The requirements should be compat-
ible with the funds available for the survey.
The target response rate for the survey as a
whole.
(2) Concise technical proposal instructions.
This supporting document should include any special
information you want from the offerors to help in
evaluating their technical capabilities. At a
minimum, you will want to instruct the offerors
to provide information on --
Their survey research experience, particularly
experience with and typical approaches to --
-- Sampling frame refinements;
-- Probability sample designs -- especially for
large-scale surveys;
-- Data collection methodology (including follow
up procedures);
-- Questionnaire development and testing;
-- Estimation procedures, including nonresponse
adjustments;
-- Variance computation procedures -- especially
for complex samples;
-- Training and supervision (of interviewers,
coders , etc.);
-- Data analysis in large-scale surveys;
-- Data processing;
-- Quality control; and
-- Publications.
Their relevant prior work, including examples
of questionnaires, sampling documentation, in-
terdiewing manuals, and survey reports prepared
by the firm. Ideally, you should request in-
formation on all previous work pertaining to
-42-
-------
the subject areas of concern -- not just work
the offerors want to submit -- over some reason-
able time-frame (say, five years). (The project
staff should verify at least a sample of the
work cited in the proposals before the discus-
ions/negotiations with the offerors begin.)
Their staff experience, including the names and
resumes of key staff members. It may be desir-
able to include a "key personnel" clause stating
which staff members cannot be replaced without
prior approval by the Agency.
The consultants they plan to use. (Request of-
ferors to include a letter of intent from any
key people who are not permanent staff members.)
The background and experience of their inter-
viewing staff and the qualifications of super-
visory and clerical field staff.
You may want the offerors to provide a work break-
out indicating exactly what technical services will
be performed at each phase of the survey. A break-
out is extremely helpful in monitoring and control-
ling the contractor's progress.
3. Competitive Evaluation Criteria
A document specifying the measurement standards the
Agency will use to assess the technical merits of
each offeror's proposal uniformly and consistently
is essential to support a competitive procurement.
The project officer (and other members of the Techni-
cal Evaluation Panel, if any) are responsible for
developing both the technical evaluation criteria and
a scoring plan. The criteria will also be reviewed
by the contracting officer.
Note that the RFP typically will include only a sum-
mary of the evaluation factors and the numerical
weights assigned to each, but it will give the offer-
ors a sufficient understanding of the relative impor-
tance the Agency places on each part of their techni-
cal proposals, such as (a) the technical adequacy of
the work plan, (b) organizational capability, and
(c) the technical qualifications of the staff to be
assigned to the project.
See sections A-2 and all of section B in Chapter 6 for
further information on evaluating survey proposals.
-43-
-------
Also, the Project Officer's Handbook gives detailed
instructions for developing the evaluation factors of
competitive procurements.
In this chapter we discussed the project officer's
principal responsibilities during the first phase of
the procurement process. We provided instructions
for preparing the paperwork to initiate a procurement
request for survey work, with special emphasis on
competitive solicitations.
The next chapter, we continue our discussion of the
main supporting document in the procurement request
package -- the statement of work.
-44-
-------
CHAPTER 5
PREPARING THE STATEMENT OF WORK
The statement of work for a contract survey describes the pur-
pose and objectives of the survey, the technical requirements
of the award, and the contractor's obligations. We have devoted
an entire chapter to the statement of work because it is the
most important part of the procurement request package.
As noted in the previous chapter, the project officer typically
writes the statement of work, which should be attached to the
package as a supporting document. The contracting officer
usually will include it intact as an appendix to the RFP. Later,
possibly with some modifications, it will be incorporated into
the contract.
This chapter discusses --
The importance of the statement of work to
the successful outcome of the project;
The specific topics the sponsoring office
should cover in the statement of work; and
Tips on compiling the statement.
Our discussion here focuses on the proper content of a statement
of work for a competitive solicitation, but many parts are
applicable to other procurement options. A statement of work is
an essential ingredient of any survey contract.
A. IMPORTANCE OF A WELL-DEFINED WORK STATEMENT
The importance of preparing a work statement that clearly
defines the technical requirement of the proposed project
cannot be over-emphasized. The statement should be precise
enough to allow prospective contractors to propose intelli-
gently, confidently, and competitively, but not so defini-
tive it frightens away qualified competitors.
A work statement that is neither too restrictive nor too
broad generally has many positive effects. It will --
-45-
-------
Increase the number of qualified responses to the RFP.
If the statement of work is ambiguous or too broad,
qualified firms may elect not to propose because the
risk of misinterpreting the Agency's requirements is
so great they will not exert the effort necessary for
a sound response. Conversely, if it is too restric-
tive, some capable firms may not propose, fearing the
Agency will over-control their performance.
Improve the quality of the proposals.
If the project staff has highly qualified statistical
staff and knows exactly what type of technical assis-
tance it wants, a detailed work statement is sometimes
appropriate. Maximum specification of the survey de-
sign will communicate exactly what the Agency expects
from the offerors, but it also allows the offerors to
reiterate the Agency's stated requirements. An in-
flexible work statement also tends to discourage any
attempt to improve on the stated requirements.
Hold down the cost of the survey.
The offerors' perceptions of the risks involved in
undertaking the work outlined in the RFP may affect
the contract price.
If the work requirements are well defined and suffi-
ciently flexible to permit offerors to demonstrate
their understanding of the research problem and their
ability to help the Agency resolve it, negotiating
a firm fixed-price contract may be possible. This
type of contract requires less administrative over-
sight by the Agency and thus lowers the total cost of
the project.
An ill-defined work statement, on the other hand,
usually makes it necessary for the Agency to use a
more expensive alternative -- a cost-reimbursable con-
tract -- because neither the Agency nor the contractor
can estimate the costs precisely enough to agree on
an acceptable fixed price.
Facilitate the evaluation process.
If the scope of work and the scoring plan are clearly
defined in the RFP, it is much easier for the Agency
to evaluate the proposals fairly. Offerors also are
less likely to misinterpret the Agency's requirements.
-46-
-------
SUGGESTED CONTENT OF THE STATEMENT OF WORK
The next thirteen pages discuss the topics the sponsoring
office should cover in the statement of work. There are 24
topics (shown in Exhibit 5) organized into four categories:
(1) Purpose, objectives, and background of the study;
(2) Design specifications;
(3) Data collection requirements; and
(4) Data processing, tabulation, and analysis
requirements.
We recommend that the project officer -- or whoever writes
the statement of work for the procurement request package
-- include some information under each topic to help pros-
pective offerors prepare their proposals. This means --
= Specifying the Agency's requirements for the particu-
lar topic; or
Asking the offerors to state how they intend to deal
with the topic if awarded the contract; or
Both.
Content for each of the 24 topics is suggested below.
1. Purpose, Objectives, and Background of the Study
The topics in the first category are (1) purpose of the
survey, (2) research objectives, and (3) background
material.
(1) Purpose of the study.
Succinctly state the reasons the Agency is under-
taking the survey. Indicate whether the substan-
tive findings are to be definitive or whether the
survey is intended as a pretest, pilot test, or
feasibility study. State how the survey results
are intended to support program goals and cite
any relevant statutes. If appropriate, briefly
discuss the research problem in the larger con-
text of the Agency's mission, particularly if the
survey is part of an ongoing research program.
(2) Research objectives.
Define the research objectives as clearly and
precisely as possible in terms of the information
to be collected and the hypotheses to be tested.
-47-
-------
EXHIBIT 5
TOPIC GUIDE FOR THE STATEMENT OF WORK
1.
PURPOSE, OBJECTIVES, AND BACKGROUND OF THE STUDY
(1) -
(2) -
(3) -
Purpose of the study
Research objectives
Background information
2 .
STUDY
DESIGN SPECIFICATIONS
(4) -
(5) -
(6) -
(7) -
(8) -
(9) -
Target population and coverage
Target response rate
Required use of probability sampling
Level of precision (sampling error)
Sampling plan
Sampling frame
3.
DATA
COLLECTION REQUIREMENTS
(10) -
(11) -
(12) -
(13) -
(14) -
(15) -
(16) -
(17) -
(18) -
(19) -
(20) -
(21) -
Collection methods
Ques tionnaire(s)
Pretes ts
Data rights
Respondent's obligation to reply
Confidentiali ty
Incentive payments to respondents
Advance notice to respondents
Length of interviews
Interviewer qualifications
Interviewer training
Quality control of the data collection
4.
PROCESSING, TABULATION, AND ANALYSIS REQUIREMENTS
(22) -
(23) -
(24) -
Quality control of the data processing
Data descriptions
Tabulation and analysis
-48-
-------
If possible, include the key study variables.
(See our discussion of this topic in section A-l
of Chapter 3.)
(3) Background information.
Discuss any available reference material or data
that would help prospective offerors develop their
proposals, e.g., studies done by the Agency (or
others), sampling frames, estimates of the number
and geographic distribution of the target popula-
tion, findings of related research.
(If these materials are not too bulky, they can be
included as attachments to the RFP. Otherwise,
indicate when and where offerors can obtain them.
Do not include material that would be of interest
only to the successful offeror.)
2. Survey Design Specifications
Of the six topics in this category, four are among the
minimum design specifications discussed in Chapter 3,
namely, (4) target population and coverage, (5) target
response rate, (6) required use of probability sam-
pling, and (7) level of precision (or sampling error).
The two additional related topics are (8) the sampling
plan and (9) the sampling frame.
For additional information on all of these topics, see
section A-l of Chapter 3.
(4) Target population and coverage.
Define the survey population and clearly specify
the reporting units (individuals, households, bus-
inesses, or industries) from which the information
is to be collected. Allow offerors the option of
describing alternate units, if appropriate.
Identify the geographic areas to be covered and
any subpopulations of special interest to EPA.
Clearly indicate any exclusions in coverage.
(5) Target response rate.
Specify a minimally-acceptable target response
rate for the survey as a whole and the method
offerors should use to calculate it.
As discussed in Chapter 3, many factors have to
be considered in determining an acceptable target
-49-
-------
rate. Generally speaking, it should not be less
than 75 percent (of those eligible to be in the
target population -- not just those at home when
the interviewers call, for example). A survey
expert can help you determine an appropriate rate.
A low response rate will do more to render the
survey results questionable than, say, using a
sample that is too small. This is because there
is no way of making valid inferences from the
portion of the population that does not respond.
If the response rate in a large survey is low,
there is an even greater chance of bias.
Adequate follow-ups of nonrespondents are essen-
tial to achieve whatever target rate you set.
Ask offerors to specify what they will do to meet
the Agency's target. (See Chapter 5 of Volume II
for information on establishing quality assurance
procedures)
Required use of probability sampling.
State that probability sampling must be used at
each stage of respondent selection. You may
encounter difficulties in clearing the survey
through OMB if you do not insist that probability
selection methods be used. OMB requires a special
justification if the Agency specifies another se-
lection method. (For more information on sampling,
see Chapter 4 of Volume II.)
Except in highly unusual circumstances or for a
pretest, we recommend that you ask the contracting
officer to insert the following clause in the RFP:
"Unless otherwise specified in the statement of
work, any offeror's response to this Request
for Proposals (RFP) shall contain probability
sampling methods for the selection of respond-
ents or subjects for any survey or other study
in which sampling from a defined population is
required. Sampling plans and cost proposals
shall be based on such methods. If, however,
an offeror feels that a nonprobability sampling
approach would be more efficient, offeror may
elect to submit an alternative proposal in
addition to the required proposal. Any offeror
not submitting a basic proposal utilizing prob-
ability sampling methods shall be considered
nonresponsive to the RFP."
-50-
-------
(7) Level of precision (sampling error).
The goal of the sample design is to get the least
sampling error and the least bias given the avail-
Table resources. For most surveys,the sample that
is ultimately conceived represents a compromise
between demands for precision and budget restric-
t ions.
As indicated earlier, it is essential for the
Agency to establish the level of precision it will
tolerate for one or more of the key statistics.
error is one of the princi
pal means of assuring
achievement of the researc
i obj ectives.
State the desired level in terms of standard error
or relative standard error. Indicate whether it
is to apply to any specific subpopulations. Be
sure to state your requirements clearly. A mis-
interpretation of the Agency's requirements for
sampling error can lead to gross differences in
the clustering and sample sizes the offerors will
propose.
The stated level of precision should be compatible
with the funds available for the survey. Ask a
survey expert to help you determine an error rate
that can be achieved given the funds available
for the project.
(8) Sampling plan.
Once you have stipulated the level of precision
in terras of some minimum allowable sampling error
for one or more of the key statistics, ask offer-
ors how they plan to achieve this level.
Instruct the offerors to base their choice of sam-
pling units, sample sizes, and selection methods
(for each stage, if applicable) on optimization
considerations -- i.e., maximum precision per unit
of cost.
More specifically, offerors should be asked to
describe and justify:
-- Their choice of the sampling units (e.g., per-
sons, households, counties, enumeration dis-
tricts, businesses, industries).
-51-
-------
The Agency can designate the units for some
stages of selection such as the primary sam-
pling units (PSUs) , and ask the offerors to
define the remaining units in the sample.
For a multiple-stage sample design, it may be
useful to require offerors to use a certain
number of PSUs. Otherwise, the resulting sam-
ple might not yield acceptable results.
-- Their choice of sample size. As we recommended
earlier,do not specify the initial sample size.
It is not desirable or practical for EPA to
specify both the sample size and a minimally-
acceptable level of precision. The reason is
it may not be possible to achieve the desired
level of precision using a sample of the pre-
scribed size, given the resources available for
the work. You may require a minimum sample
size for specific subgroups of special concern,
however.
-- The methods they propose to use to select units
from the frame (at each stage, if a multiple
stage sampling plan is to be used).
-- Any variation from strict probability sampling
the offerors consider necessary to deal with any
special problem such as unavoidable coverage
deficiencies.
-- The methods they propose to use to control and
measure biases and other nonsampling errors
during the data collection and data processing.
(9) Sampling frame.
An accurate, up-to-date, complete frame is essen-
tial if the sample is to represent the target pop-
ulation adequately. Indicate whether the Agency
will provide the frame -- the sources from which
the sample members will be selected -- or whether
the contractor is to develop it. You may recommend
certain sources.
Ideally, the sampling frame should fully cover the
target population. Therefore, be sure to indicate
any areas or subgroups that should be excluded,
i.e., Alaska, Hawaii, military bases.
If the Agency is to supply the frame, make sure
(a) there are no legal or regulatory restrictions
-52-
-------
preventing its use, such as confidentiality provi-
sions in the Federal Privacy Act or environmental
statutes; (b) it is current, complete, and con-
tains no duplications; and (c) you can get it to
the contractor in a usable format in time to meet
the project schedule.
On the other hand, if you want the contractor to
supply the sampling frame, ask offerors to provide
information on how they propose to construct the
frame and ensure its accuracy.
3. Data Collection Requirements
The next twelve topics relate to the data collection
methodology, specifically to requirements for (10) col-
lection methods, (11) questionnaire(s), (12) pretests,
(13) data rights, (14) respondent's obligation to re-
ply, (15) confidentiality, (16) incentive payments to
respondents, (17) advance notice to respondents, (18)
length of interviews, (19) interviewer qualifications,
(20) interviewer training, and (21) quality control.
(10) Collection methods.
Ask offerors to justify the methods and techni-
ques they intend to use to collect the data. You
may suggest an acceptable data collection method
and explain any time constraints on the data
collection phase, but we recommend you do not
specify the collection methodology unless you
have good reasons for doing so. (See Chapter 2
of Volume 11 for more information.)
(11) Questionnaire(s).
Include a draft questionnaire, if possible. It
gives the offerors a basis for estimating the
costs of collecting and processing the data.
Otherwise, include a list of suggested questions
or state EPA's information requirements in suffi-
cient detail to allow offerors to construct a
preliminary questionnaire and determine the amount
of effort required to polish and test it.
Indicate whether open-ended or precoded, multiple-
choice answers are to be used, and whether re-
sponses will be mandatory or voluntary.
Also indicate any special requirements such as
(a) a Spanish-language version in areas where
-53-
-------
Spanish is the predominant language, (b) differ-
ent versions of the questionnaire to bring out
local concerns in various regions, or (c) supple-
mental diaries.
State how many copies of the questionnaire will
be required and who is to reproduce them.
The contractor should have a major role in the
development of the final questionnaire, but the
Agency should retain review authority. (We recom-
mend that you have an Agency survey expert review
the final questionnaire. Unless the questions
are properly phrased, the resulting data are apt
to contain serious biases.)
(See Appendices B and C for examples of question-
naires used in recent EPA surveys. Also see Chap-
ter 3 of Volume II for information on the entire
questionnaire-development process.)
(12) Pretests
A pretest or pilot test is the only way of making
sure the survey is workable. A pilot test repli-
cating the proposed methodology as closely as
possible is vital for many surveys. (A pretest
assesses the validity of the questionnaire only,
as a rule.)
If you intend for the contractor to pretest the
survey, ask offerors to propose:
-- The size of the test sample. Pretest or pilot
test samples should be large enough to bring
out any significant problems with the question-
naire (or the proposed data collection tech-
niques) .
OMB clearance is not required if fewer than ten
respondents are used. However if (a) the sam-
ple is very large, or (b) the questionnaire is
complicated and lengthy, or (c) the target pop-
ulation is very heterogeneous on key character-
istics, a test population this small is not
adequate. Although an additional clearance
will be required if ten or more respondents are
used in the pretest, the review often has sig-
nificant benefits. It can reveal overlap with
ongoing data collection activities in other
agencies or ways of improving the proposed
methodology.
-54-
-------
-- A method for selecting the respondents. While
probability sampling is important for choosing
the respondents in the survey proper, it usual-
ly is not necessary for a pretest. One or more
"purposive" samples are frequently chosen for
a pretest to make sure that advance information
on key subpopulations is obtained.
-- The timing of the pretest. Offerors should
allow enough time between the pretest and the
actual survey to analyze the results and amend
the questionnaires (and methodology) according-
ly.
-- Ways to increase the utility of the survey
results. For example, consultations with po-
tential data users often provide valuable tips
that will improve the questionnaire. Users
also may comment on the degree of precision
they would like to see and the timing of the
survey.
(13) Data rights.
Discuss (a) how the rights of data subjects or re-
spondents are to be protected, (b) Agency policies
for monitoring interviews, and (c) policies gov-
erning the use of the survey data. Offerors should
be required to assure respondents that their in-
terests will be protected to the maximum possible
extent.
If the data are being collected for general sta-
tistical purposes, the Agency usually can advise
respondents that the information they supply will
be used only for statistical tabulations and that
individual replies will be kept confidential. If
not, the respondents should be advised of the ex-
tent of disclosure, how the Agency intends to use
the data, and who will be permitted access to it.
(14) Respondent's obligation to reply.
Indicate whether responses will be voluntary or
mandatory. The Paperwork Reduction Act also re-
quires that questionnaires include information on
the respondent's obligation to reply.
(15) Confidentiality.
Discuss confidentiality and ask offerors to state
the measures they intend to take to protect the
-55-
-------
information the respondents supply.
Confidentiality is of prime concern to all repu-
table survey research organizations. However,
if the Agency promises confidentiality, the con-
tractor raay have to take special precautions
(e.g., lock up data and allow access only by
authorized personnel).
Consult an attorney in the Office of General Coun-
sel (OGC) about the extent to which the Agency
raay pledge confidentiality and the measures that
should be used to safeguard respondents' data.
This should be done before the questionnaire is
submitted for OMB clearance.
(16) Incentive payments to respondents.
Advise offerors if the respondents should be
financially compensated for completing the ques-
tionnaire and explain why this is necessary.
Incentives do not always contribute to a higher
response rate, but they may be crucial to the suc-
cess of a lengthy or complex survey. Completing
or responding to a questionnaire may require a
significant amount of work or time on the part of
the respondents. If replies are voluntary, con-
sider compensating respondents for their time.
If you do plan to pay the respondents, get the
approval of your Division Director. You also
will have to explain these payments in the OMB
clearance request. If the success of the survey
depends on compensating respondents, discuss this
with the Information Management Branch early on.
(17) Advance notice to respondents.
The response rate of many surveys can be consider-
ably improved if the respondents are given advance
notice of the Agency's intent to collect certain
information. This gives them an opportunity to
review their records and documents before they are
are interviewed and speeds up the process.
If you intend to have the contractor handle the
advance notifications, advise offerors of this
requirement so they can plan their resources
accordingly. (For additional information on all
aspects of interviewing, see Chapter 5 of Volume
II.)
-56-
-------
(18) Length of interviews.
Specify a time limit or ask offerors to propose
a ceiling on the length of the interviews. Some
respondents will not cooperate if the interview
is too long.
If you let the offerors know approximately how
many questions you intend to ask.it will be easier
for them to propose the interview length. General-
ly speaking, face-to-face interviews should not
last more than 60 minutes; 30 minutes is the upper
limit for a telephone interview. For a voluntary
household survey, 120 minutes is the maximum time
you can expect respondents to devote to a mail
questionnaire. For an industrial survey, where
the Agency can require mandatory responses, how-
ever, there is no need to limit the length of the
interviews.
(19) Interviewer qualifications.
Indicate any special qualifications the interview-
ers should have, such as expertise in a specific
field of study, knowledge of a foreign language,
or previous interviewing experience with certain
types of respondents (e.g., corporate executives).
(20) Interviewer training.
Ask offerors how they propose to train the inter-
viewers (e.g., printed interviewer guides, train-
ing conferences), and state whether the Agency
will participate in or observe the training.
(21) Quality control
Quality control of all data collection operations
-- checking sample selection.verifying interviews,
etc. -- is vital to ensure that the survey design
is faithfully implemented. However, we recommend
that you do not stipulate any particular quality-
control procedures. State only that quality con-
trol is a necessary element of the survey and ask
the offerors to say how they propose to control
the quality of the data collection.
4. Data Processing, Tabulation, and Analysis Requirements
Specify or require information from the offerors on all
applicable aspects of the data processing and analysis
-57-
-------
operations (e.g., procedures for editing and coding,
keying instructions for processing personnel, verifi-
cation procedures, any required tabulations, weighting
for nonresponse and missing data, and any analysis
requirements).
At a minimum, the following topics -- (22) quality con-
trol, (23) data descriptions, and (24) tabulations
and analysis -- should be covered in the statement of
work.
(22) Quality control of data processing.
We recommend that you do not specify any particu-
lar quality control techniques for the data pro-
cessing operations. Rather, ask offerors to state
how they propose to control them, e.g., checking
the editing and coding of the responses.
(23) Data descriptions.
Specify the format of all data tapes, cards, and
printouts the contractor is to supply. For ex-
ample, indicate if the data tapes must be compat-
ible with EPA computers.
(24) Tabulation and analysis.
If possible, indicate the form and content of the
tabulations. Otherwise, ask the offerors to supply
table shells.
If you wish the contractor to analyze the data,
specify in what form you want them presented. (For
additional information on preparing an analysis
plan, see Chapter 1 of Volume II. For more infor-
mation on data processing, see Chapter 6 of that
volume.
C. TIPS ON COMPILING THE WORK STATEMENT
Before you begin to write --
= Review all applicable laws and Agency regulations,pro-
cedures, and policies and discuss them, as appropriate,
with other program officials, your contracting officer,
your Information Management Coordinator, and your rep-
representative in the General Counsel's Office. This
will allow you to make the offerors aware of any appli-
cable environmental statutes and Federal laws that may
-58-
-------
impose constraints on the survey, such as the Privacy
Act and the Paperwork Reduction Act.
Examine other Agency work statements,solicitation doc-
uments (RFPs), and contracts dealing with similar re-
search. Query staff who have worked on these projects
about problems they encountered and ask thera how you
can avoid them in your survey.
While you're writing --
= Be sure to include all technical requirements and con-
tractor obligations. Surprising offerors during the
negotiations with changes in the scope of the work may
cause serious problems, particularly if the changes
affect the schedules or budgets they have proposed.
= Use illustrations to enhance offeror understanding of
technical provisions.
= As for language --
Be as concise and clear as possible. Use accu-
rate, consistent terminology. Avoid words with
more than one meaning, jargon, and pronouns with
vague antecedents.
Use mandatory language properly. Use "shall" to
express a binding provision in a work statement.
Use imperatives for obligatory requirements
e.g., "The following areas shall be covered in
the survey..."
Before sending the document to your supervisors for review
Carefully edit the text. Check for unnecessary dupli-
cations, contradictions, typing errors, omissions, etc.
= Have a statistician or sampling expert review the sur-
vey design specifications.
-59-
-------
This chapter presented a topic guide to help in drafting
the technical content of the statement of work for a
typical contract survey. The guide includes 24 topics,
all of which should be covered to help the prospective
offerors prepare their proposals.
We also discussed the positive effects of a well-defined
work statement on the response to the RFP, the evaluation
of the proposals, and the price of the contract.
The next chapter examines the project officer's role in
selecting the contractor, especially in evaluating the
technical proposals.
-60-
-------
CHAPTER 6
SELECTING THE CONTRACTOR
This chapter examines the project officer's role in the next
phase of the procurement process -- the evaluation of the offers
received in response to the RFP and the selection of a qualified
firm to carry out the survey. More specifically, it explains --
The project officer's administrative and
advisory responsibilities in evaluating the
proposals and selecting the most qualified
offeror; and
Tips on evaluating the technical proposals.
A. THE PROJECT OFFICER'S ROLE IN SELECTING THE CONTRACTOR
After the deadline for receipt of the proposals and until
the contract is awarded, the project officer has many im-
portant advisory and administrative responsibilities such
as (1) safeguarding information from the offerors, (2)
evaluating the technical proposals, (3) and assisting the
contracting officer in the discussions or negotiations with
the offerors in the competitive range. These activities
are briefly described below.
Keep in mind that, as in all phases of the procurement
process, the Procurement and Contracts Management Division
(PCMD) plays the lead role in the source evaluation and
selection process.
For all competitive procurements in excess of $25,000, the
PCMD official in charge of this phase -- the Source Selec-
tion Official -- is responsible for (a) appointing the mem-
bers of the Source Evaluation Board (if convened), the
Technical Evaluation Panel, and the Business Evaluation
Panel; (b) overseeing all administrative and procedural
aspects of the procurement; (c) making final decisions on
the competitive range and selection of the contractor; and
(d) approving the contract document. If the award is ex-
pected to be more than S5 million, the contracting officer
will serve as the Source Selection Official -- unless the
-61-
-------
the Chief of the Contracting Office designates someone else
to serve in this capacity. (For more complete information
on PCMD's role in the evaluation and selection process, see
Subpart 1515.6 of the EPA acquisition regulation.)
1 Safeguarding Information from Offerors
Throughout the source evaluation and selection phase of
the survey, all Agency personnel must be careful to
comply with Federal policies governing the handling and
disclosure of information about the procurement. These
policies are designed to prevent any offeror from gain-
ing an unfair competitive advantage in the awarding of
a contract.
Therefore, as project officer, remind your staff --
To keep all information received from offerors in
strict confidence. This means taking proper care
in handling the proposals and any supplementary
information the offerors may submit for clarifica-
tion purposes prior to the award.
To disclose no information about the pending procure-
"r ment to people outside the Agency. Only authorized
contracts personnel may do this.
To refer all correspondence or inquiries about the
survey, the RFP, and the status of the evaluation or
the negotiations to the contracting officer. All
correspondence sent to the offerors must be signed
by the contracting officer.
2. Evaluating and Scoring the Proposals
The sponsoring office's principal impact on the selec-
tion of the contractor is in performing the technical
evaluation of the offers received in response to the
solicitation (the RFP). The project officer always
chairs the Technical Evaluation Panel, which has sole
responsibility for reviewing and scoring the offerors'
organizational and staff capabilities and the technical
adequacy of the proposed work plans. The Source Selec-
tion Official (usually the contracing officer), on the
other hand, determines the composition of the Technical
Evaluation Panel.
If the award is expected to be $500,000 or less, the
project officer may be the only member of the panel.
On awards over $500,000, the panel also may include
-62-
-------
technical experts from inside and outside the Agency,
statisticians, and perhaps someone from the target
population.
As soon as the deadline for receipt of the proposals
expires, the contracting officer (or another contract
specialist delegated by the Source Selection Official)
will forward copies of the technical proposals to you.
The panel must evaluate and score them according to
the criteria and scoring plan described in the RFP.
Each member of the Technical Evaluation Panel must in-
dependently evaluate and score each proposal. If the
procurement is valued at more than $500,000, you (as
panel chairperson) must develop a consensus opinion on
the scores the panelists assign to each offer. You
may not average the individual panelists' scores to
arrive at an overall score for the panel as a whole.
When the panel completes its evaluation of the offer-
ors' initial proposals, you must submit a technical
evaluation report to the contracting officer. The
report must include --
A detailed scoring of each offer received, and a
narrative summary of facts and findings regarding
the significant strengths, weaknesses, and risks
associated with each offer. For procurements valued
at $500,000 or less, you may use the two-page EPA
Form 190061, "Proposal Technical Evaluation, to
report on each offeror (see Exhibit 6.) A separate
page 2 should be used for each evaluation criterion.
Any clarification of the technical proposal you want
the contracting officer to request from the offerors.
A statement that the panel members have no (actual
or potential) personal conflicts of interest.
Information indicating an actual or potential organ-
izational conflict of interest on the part of any
offeror.
The panel's report also should include --
Any suggested modifications in the statement of
work to be discussed with the offerors.
A justification of any recommendation to eliminate
certain offerors from further consideration.
The contracting officer will examine the scores and
other information in the technical evaluation reports
-63-
-------
A EPA
US ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON. DC 20460
PROPOSAL TECHNICAL EVALUATION
RFP NUMBER AND TITLE
OFFERER
EVALUATED BY
DATE
MAXIMUM SCORE ATTAINABLE
EVALUATION SCORE
GENERAL COMMENTS
EPA Form 1900-61 (2-84)
CONTINUE ON REVERSE
-------
EXHIBIT 6
TITLE OF CRITERION
ASSIGNED WEIGHT (in points)
Val ue
Descriptive Statement
The element is not addressed, or is totally deficient and without merit.
20
The element is addressed but contains deficiencies that can be corrected only by changes to relevant
portions of the proposal.
40
Clarification is required. Final scoring of the element will be made following limited discussions or full
negotiations.
60
The proposal element is adequate. Overall it meets specifications.
70
3.5
None
80
The proposal element is good with some superior features.
90
4.5
None.
100
The proposal is superior in most leatures
SCORE <% of Assigned Weight)
COMMENTS
EPA Form 1900-61 (2-04) REVERSE
-------
price and other factors, determine which offerors are
within the competitive range and thus eligible for
further consideration.
3. Supporting the Discussions and Negotiating with Offer-
ors in the Competitive Range
Once the Source Selection Official approves the selec-
tion of the offerors in the competitive range, the
contracting officer will begin to conduct limited cost/
technical discussions with these firms. The discus-
sions (or negotiations) may consist of a single phone
call or a series of face-to-face meetings. As the
procurement dictates, the discussions will culminate
with in-depth negotiations with one or more firms, or
complete cost and technical negotiations with all the
offerors in the competitive range.
Throughout this period, the contracting officer may con-
sult you (and other members of the Technical Evaluation
Panel, perhaps) about cost and price matters as well as
technical concerns. Although the technical panel serves
primarily in an advisory capacity, you may be asked to
participate in the discussions/negotiations if --
The Source Selection Official determines that the
complexity and proper interpretation of the Agency's
requirements necessitate expert technical partici-
pation ;
Further clarification of the revised proposals is
necessary;
There is a need for an explanation of any technical
deficiencies in the proposals or the offerors' plans
to rectify these deficiencies; or
PCMD desires a technical assessment of certain op-
tions (brought out in the negotiations) that may
reduce the cost of the contract.
If you and the other panelists do participate in these
meetings, do not to discuss the ideas or approaches of
any of the offerors still in contention.
You may also have to prepare supplemental technical
evaluation reports of any revised proposals the offer-
ors submit at the Agency's request.
Finally,you will have to deliver a supplemental eval-
uation report to the Source Selection Approval Official
-66-
-------
once you have completed your assessment of the revised
proposals and prior to the final selection. A supple-
mental report must be prepared for each offeror in
the competitive range.
The report should include --
= A discussion of whether or not the cost proposal
adequately reflects the substance of the technical
proposal and the requirements of the solicitation;
A summary evaluation of the offeror's technical per-
formance on recent or current projects; and
= Any changes to the initial technical scores and a
narrative evaluation, based on discussions (or nego-
tiations) and the revised technical proposal.
The contracting officer will review the final technical
evaluation reports and prepare a source selection deci-
sion document outlining the consideration given to
price, technical merit, and other factors such as the
offerors' prior performance, subcontracting plans, etc.
The contracting officer (or other contract specialist
serving as the Source Selection Official) then will
make a final determination on the contract award.
B. EVALUATING THE TECHNICAL PROPOSALS
Because the project officer has the lead role in the eval-
uation of the technical proposals, the remainder of this
chapter is devoted to suggestions for weighing (1) the
overall technical quality of the initial proposals; and the
technical adequacy of the (2) survey design, (3) work plan,
and (4) organizational and staff capabilities of each offer.
1. General Evaluation Considerations
First, we'll discuss some overall evaluation considera-
t ions.
Assuming that the Agency's technical requirements were
well defined, each offeror should demonstrate a clear
understanding of the scope of work outlined in the RFP
and possibly further clarified at one or more prepro-
posal conferences.
A misinterpretation of one or more of the Agency's
requirements generally indicates a weakness in the
-67-
-------
proposer's technical capabalities. Of course, if most
of the offerors misinterpret the same requirements,
you probably did not clearly state them in the RFP.
Note that if the Agency failed to include a key re-
quirement -- that probability sampling must be used,
for example, the proposal cannot be low-rated solely
because the offeror failed to account for the omission.
If the Technical Evaluation Panel determines that the
Agency's failure to adequately define or include a
certain key requirement may seriously prejudice the
survey outcome, the panel can ask the contracting offi-
cer to amend the RFP and receive revised offers. This
will delay the project considerably but it may be the
only way to ensure that the research objectives can be
ach ieved .
To the extent the design specifications in the RFP per-
mit, the overriding consideration in evaluating the
technical proposals should be the relative efficiency
of the proposed survey designs.
From the standpoint of technical adequacy, the most
"efficient" design is the one proposing to provide sta-
tistics having the highest level of precision (the
least sampling error). The "cost" of this efficiency
must be taken into account later in the course of the
selection process.
2. Evaluating the Key Design Elements
As we have emphasized throughout this handbook, the
successful outcome of the survey depends largely on
the Agency's specifications for certain key design
elements. In evaluating the technical adequacy of the
offers, therefore, it is important to pay special at-
tention to how the offerors propose to meet EPA's re-
quirements governing (a) target population and cover-
age , (b) target response rate, (c) use of probability
sampling, and (d) level of precision (sampling error).
Assuming you clearly specified the Agency's require-
ments concerning these four key design elements, some
of the major considerations for evaluating them are --
Target population and coverage.
Look for any exclusions in coverage that were not
expressly discussed in the RFP.
-68-
-------
For example, in order to propose a low per-unit
cost, some offerors may exclude Hawaii or Alaska
(from a "national" survey) or certain "difficult-
to-get" subpopulations such as non-English-speaking
households or institutionalized people. If the RFP
did not specify these exclusions, you will have to
weigh the advisability of allowing them. If the
Agency decides to allow any exclusions proposed by
an offeror, the others must be given an opportunity
to amend their proposals accordingly.
Target response rate.
Assuming the RFP established a minimally-acceptable
response rate for the overall survey and asked the
offerors to indicate how they plan to meet this tar-
get rate --
(1) Carefully scrutinize the offerors' plans for
following up the interviews of the initial sam-
ple . Do not give an unfair advantage to offer-
ors who propose a minimal or vaguely defined
follow up in order to come up with a low offer.
For example, if one of the offerors proposes
to substitute respondents after a certain num-
ber of unsuccessful attempts to collect data
from the the initial sample, both the precision
of the statistics and the cost of the survey
will be affected. Substituting respondents wil1
permit the offeror to achieve the Agency's tar-
get rate at a lower cost, but the likelihood of
bias is higher.
If the RFP specified the number and timing of
follow-ups in nonresponse situations, check to
see if the Agency s requirements were correctly
interpreted. Any misinterpretation of these
requirements may indicate technical weakness.
Failure to allow sufficient staff time for the
follow-up work also is indicative of technical
inadequacy. (See Chapter 5 of Volume II for more
information on follow-up interviewing.)
(2) Carefully examine the proposed data collection
procedures to see if they are are likely to
elicit adequate responses from key subgroups
in the target population. For example, for some
surveys it may be important to provide bilingual
interviewers or interviewing materials to avoid
underrepresentation of particular ethnic groups.
-69-
-------
(3) Examine the response rates offerors achieved in
previous surveys to see if they attained the
target rates they initially proposed. Also
check the quality control procedures used to
achieve those rates (e.g., use of advance let-
ters to respondents, respondent incentives, and
follow-ups by trained interviewers). Careful
attention to quality control is a good barometer
of technical competence.
Use of probability sampling.
Assuming the RFP required the use of probability
sampling throughout the selection process (except
in the pretest, perhaps), check for --
(1) Any planned substitutions for nonresponse situ-
ations . Substitution generally is not an accept-
able method for reducing the nonresponse rate.
Any departure from full probability sampling in
selecting respondents during the survey tends to
introduce bias, especially in larger samples.
(2) Any departure from probability sampling in se-
lecting individual respondents within sampling
units (e.g., households, businesses). To save
time and money, some offerors may plan to allow
interviewers to use their own judgment in selec-
ting the individual respondents within each sam-
pl ing unit. This generally should not be allowed
because it tends to produce bias.
(3) Vague descriptions of the procedures to be used
for selecting respondents. The proposals should
give a step-by-step description of the proce-
dures that will be used to ensure a full proba-
bility sample. If a proposal says something
like "200 counties will be selected..." and does
not specify that the selection will be made with
"equal probability" or with "probability propor-
tionate to size," the Agency cannot determine
whether the offeror intends to use probability
selection techniques. More importantly, it will
be impossible to determine the efficiency of the
proposed design.
Be especially wary about buying into an existing
sample. Some survey research firms maintain general-
purpose, multistage household samples and may pro-
pose to use one of them. Or an offeror may propose
to draw the sample from a commercially-available
list of businesses or industries.
-70-
-------
If any offeror proposes to use an existing sampling
frame
(1) Have a sampling expert review (a) the design
specifications, (b) the actual worksheets to be
used in selecting the sample, and (c) other rel-
evant materials.
(2) Review the suggested modifications in the frame
carefully. The proposal should indicate how
(a) the offeror intends to ensure the complete-
ness of the frame, (b) new units will be added,
(c) inactive units will be eliminated, and (d)
how "foreign" or ineligible units will be iden-
tified.
(3) Make sure there are no restrictions on the use
of the sampling frame that would prohibit the
Agency from publishing information about the
sample.
Level of precision (sampling error).
Assuming the RFP required a certain level of preci-
sion (or margin of error) for one or more of the
key statistics, make sure offerors have correctly
interpreted tKe requirements. Any misintermisinter-
pretation of these requirements is likely to indi-
cate a weakness in the offerors' technical capabil-
ities .
If you asked the offerors to state how they plan to
carry out the data collection and data processing
operations, much can be gleaned about a offeror's
statistical competence by reviewing their proposed
"estimation procedures." These are procedures used
to estimate the statistics (means, proportions,
etc.), deal with missing data, and control sampling
errors. Your review should focus on whether the
proposed estimation formulas --
(1) Take proper account of the sample design and se-
lection probabilities of individual units; and
(2) Make full use of ancillary data known to be
available for all members of the target popula-
tion, whether or not they are included in the
sample.
We strongly recommend that you have a sampling expert
evaluate the proposed estimation procedures. Also,
-71 -
-------
see Chapter 4 of Volume II for further information on
all aspects of sampling.
3. Evaluating Key Aspects of the Work Plan
Offeror responses to the following topics in the state-
ment of work are also highly indicative of their tech-
nical competence: (a) the data collection methodology,
(b) the pretest provisions, and (c) the quality control
procedures proposed for both the data collection and
data processing activities.
Data collection methodology.
The methods and procedures the offerors propose to
use for collecting data generally are a good indi-
cator of technical competence.
Some of the thing9 to look for are --
(1) The appropriateness of the proposed data col-
lection method (face-to-face or telephone Tn-
terviews, mail questionnaires or a combination
of methods). There are real limits on the type
and amount of information that can be success-
fully collected with different methods. For
example, personal interviews are not appropriate
for getting detailed answers to complex techni-
cal questions if respondents must consult their
records in order to provide adequate answers, er-
Face-to-face interviews also are contraindicated
if time and cost are prime considerations; on
the other hand, this collection method is highly
appropriate for clarifying and probing respond-
ents' knowledge and experiences.
Telephone surveys are usually more difficult to
design than face-to-face surveys. Depending on
the subject matter, a phone interview may re-
quire a very elaborate script. Interviewers have
to be able to anticipate all possible answers in
order to pursue a given line of questioning --
like a flowchart, if Yes..., if No.... If this
this method is proposed, carefully examine the
"script." (See Appendix C for an example of a
telephone interviewer's guide.)
Moreover, if the sampling units are households,
offerors should include provisions for covering
households without telephones or those with un-
listed numbers.
-72-
-------
Random-digit dialing is a frequently proposed
technique for selecting respondents for a tele-
phone survey. The advantages of this technique
are (a) its relatively low cost, (b) it does
not exclude unlisted numbers, (c) it permits
good control over the interviewing process, and
(d) it gives quick results. It has two disad-
vantages: The response status or units that do
not reply is difficult to determine, and commer-
cial units cannot be distinguished from residen-
tial units. Therefore, offerors proposing to
use this technique should demonstrate an aware-
ness of its pros and cons, and indicate how they
intend to handle its limitations.
If offerors propose to use a mail questionnaire
for the survey, carefully check the procedures
they plan to use for developing the question-
naire. The length and format of the question-
naire are particularly critical for a mail sur-
vey. Generally speaking, the longer the ques-
tionnaire, the poorer the quality and quantity
of the responses. The items on a mail question-
naire, moreover, must be carefully written be-
cause no interviewer will be present to clarify
any ambiguities.
Moreover, unless the Agency has the authority to
require a mandatory response, a mail survey sel-
dom achieves an adequate response rate.
(See Chapter 2 of Volume II for further informa-
tion on the relative advantages of various data
collection techniques. Chapter III presents a
step-by-step discussion of the development of
the questionnaire.)
The appropriateness of the respondents. The
specific individuals the offerors propose to in-
terview should be capable of providing the par-
ticular information the Agency needs. For ex-
ample, if you wanted information about the num-
ber and type of employees in certain businesses,
interviewing the presidents of these firms prob-
ably would not be the best way to get it. A mail
questionnaire to the firms' personnel directors
would be more appropriate. On the other hand,
if you wanted information about the decision-
making processes of these firms, face-to-face
interviews with their chief executive officers
would be most appropriate.
-73-
-------
(3) The requirements for respondent eligibility.
In surveys involving multi-unit samples (house-
holds, for example), the "rules" governing which
individuals may be interviewed at each unit can
significantly affect the validity and cost of
the survey.
Generally speaking, if your objective is to
learn more about public attitudes and opinions,
only certain individuals within the sample units
should be allowed to respond. While respondent
rules usually permit some substitutions (e.g.,
if the intended respondent cannot be reached
after a given number of attempts, any respon-
sible adult in the same household may be inter-
viewed), a nonrandoro selection of respondents
will produce erroneous or invalid results in
certain cases. For example, a nonrandom selec-
tion of respondents for a household survey con-
ducted during the day would not truly represent
the attitudes and opinions of all the people
in the households. The reason is that members
of the household who work during the day will
be poorly represented.
Pretest provisions.
Because it is rarely possible to anticipate all the
problems and biasing effects of the proposed ques-
tionnaire and field procedures, the proposals should
include provisions for one or more pretests (or
pilot tests). This is the only reliable way of
finding out if the questionnaire is workable and, in
the case of a pilot test, if the proposed data col-
lection methodology is practicable. The principal
things to look for in the offerors' plans for pre-
tests are --
(1) Adequate representation of the survey population.
Since the purpose oT I pretest (or pilot test)
is to evaluate the workability of the actual
survey, the proposed respondents should approxi-
mate those to be used in the survey proper. Al-
though probability selection is unnecessary for
a pretest, the offerors should not propose "con-
venient" or "easy-to-get" units. Probability
sampling is generally essential for a pilot test.
(2) Size of the test sample. For a major survey,
a pretest or a pilot test using fewer than ten
respondents usually is not large enough to
-74-
-------
reveal all the problems the interviewers are
most likely to encounter in the field. To save
time and avoid the need for an additional OMB
clearance, offerors sometimes propose a test
population of only nine respondents. This should
be discouraged because a sample this small gen-
erally is not adequate to test an extremely
variable population.
(3) Adequacy of other test conditions. Since the
purpose of pretests is to anticipate and correct
problems before the survey begins, the instruc-
tions to respondents, the questionnaire, inter-
viewing techniques, the time allowed for re-
sponse, and other test conditions in the propos-
als should closely approximate those planned for
the survey proper.
(4) Sufficient time to incorporate the pretest re-
sults. A common mi stake Tn proposal schedules
is underestimating the time required to develop
a satisfactory questionnaire. Offerors should
allow ample time to correct any problems that
arise during the pretest. They also should in-
dicate that additional pretests will be carried
out if the results of the initial test are not
adequate. (See Chapter 3 of Volume II for more
information on questionnaire construction.)
Quality control procedures for the data collection
and data processing activities.
The nature and extent of the quality control proce-
dures proposed for each step in the data collection
and data processing phases are usually highly indi-
cative of technical ability. Attention to quality
control is also a good barometer of the adequacy of
the offerors' work plans in general.
At a minimum, the quality controls proposed by the
offerors shoulcl include --
(1) "Validation" of some proportion -- typically 10
percent -- of the interviews. Validation com-
monly consists of verifying that the interviews
did in fact take place and checking responses to
a few key questions.
(2) Manual checks of questionnaires for complete-
ness and proper entry.
-75-
-------
(3) Spot checks of all manual coding operations,
eluding comparisons of the results and error
rates achieved by different interviewers.
(4) Verification -- usually 100 percent -- of any
keying (card punching or key-to-tape) opera-
tions .
(5) Computer edits to detect inadmissable and out-
of-range values.
Other quality control procedures indicative of tech-
nical competence are --
(1) An objective validation of the interview pro-
cess. Some firms propose to have a subcontrac-
tor validate the data collection effort. This
is often preferable to having it done by the
contractor's own staff.
(2) Whether the proposed procedures for verifying
the data processing are "dependent" or "indepen-
dent."
Verification is considered to be independent if
the verifiers do not have access to the results
of the operation they are verifying. Errors are
are identified by comparing the original results
with those done by the verifier. Generally, in-
dependent verification is more likely to produce
accurate estimates of error rates,but it is more
cos tly.
(3) A definition of the levels of error that will be
considered acceptable and what actions will be
taken when these levels are exceeded.
4. Evaluating the Offerors' Organizational and Staff Cap-
abilities
In addition to scoring the technical adequacy of the
proposed work plans, you must carefully assess each
offeror's (a) organizational capability and (b) the
technical strengths of their key staff and consultants.
The proposals should respond to each of the items you
asked the offerors to discuss in the RFP.
The successful outcome of the survey depends as much on
the contractor's ability to execute the proposed work
plan as on the merits of the work plan itself.
-76-
-------
Some of the most important things to consider in eval-
uating each offeror1s management and technical qualifi-
cations are --
The track record of the organization and its exper-
tise in the subject area (e.g., previous experience
in contract survey management, demonstrated exper-
tise in the area of the proposed research, special
access or relationship to the target population,
references).
The adequacy of the proposed management plan. (Does
the offeror have adequate staffing and appropriate
facilities and other equipment? Are the plans for
training staff and interviewers, etc., adequate?)
The experience and technical qualifications of
key staff members and consultants. (For example,
if the survey is national Tn scope, the offeror
should have a national network of interviewers,
considerable experience in survey management, and a
diversity of technical skills.)
In this chapter we examined the project officer's role
in selecting a qualified contractor to provide the sur-
vey support services described in the RFP. In addition
to discussing the project officer's administrative and
advisory responsibilities, we made suggestions for
evaluating the technical adequacy of the proposals and
the offerors' organizational capabilities.
The next chapter examines the project officer's role in
managing the survey after the contract is awarded.
-77-
-------
CHAPTER 7
MANAGING THE CONTRACT
This chapter gives instructions for managing the survey after
the contractor is aboard. In a contract survey, effective
management of the post-award phase of the procurement is as
important to accomplishing the survey objectives as a well-
thought-out work plan.
Specifically, this chapter examines the project officer's role
in --
Generally fulfilling the Agency's contractual
commi tments ;
9 Obtaining internal and OMB approval of infor-
mation collection activities;
Monitoring and controlling the contractor's
technical performance; and
Helping to administer the contract.
A. FULFILLING THE AGENCY'S CONTRACTUAL RESPONSIBILITIES
The project officer and the contracting officer are jointly
responsible for ensuring that the survey is conducted in
accordance with the terms of the contract. Your authority
and limitations will be specified in the contract. Usually
the project officer is delegated authority for --
= Technical oversight of the contract, which involves
fulfilling the Agency's technical commitments as well
as monitoring and controlling the contractor's perfor-
mance throughout the survey; and
Certain administrative oversight activities which sup-
port the contracting officer in fulfilling the legal
and operational terms of the contract.
It is important for you to make sure the Agency fulfills
its technical commitments"? Any delays or omissions by the
project staff in maintaining EPA's contractual commitments
may prevent the contractor from legally meeting the terms
_79_
-------
of the contract and thus excuse the contractor from meet-
ing some of their obligations.
For example, if EPA is committed to supplying the sampling
frames or certain data for the analysis work and the proj-
ect staff does not provide them on time, the contractor
may not be able to meet the agreed deadlines. Agency com-
mitments to review and approve the deliverables or to ob-
tain clearances from other Federal agencies (OMB, for in-
stance) in a timely manner also force EPA to share the re-
sponsibility for the contractor's performance. Section B
below examines the sponsoring office's role in obtaining
OMB clearances.
You also must carefully monitor the contractor's progress
throughout the life of the contract and, if necessary,
control it. This does not mean you personally must over-
see all the technical provisions of the contract, but you
are responsible for seeing that they are properly develop-
ed and executed. You can obtain whatever technical assis-
tance is necessary from your staff, other Agency experts
(including statistical experts and consultants), or other
contractors. Section C below discusses these monitoring
activities at length.
Your role in supporting the contracting officer's fulfill-
ment of a host of administrative responsibilities is ex-
amined in section D below. Keep in mind that only the con-
tracting officer can make changes in the contract, either
within or outside the scope of work. Always consult with
your contracting officer when you wish to make any changes
or issue directions to the contractor that may represent
(or lead to) a change in the contract.
B. OBTAINING OMB CLEARANCES
As discussed in Chapter 3, the Paperwork Reduction Act re-
quires all Executive Branch agencies to obtain a clearance
from the U.S. Office of Management and Budget (OMB) for in-
formation collections involving ten or more members of the
public. Hence, a clearance is required for the survey and
for any pretests involving ten or more respondents. You
may request clearance for the pretest and the survey in
the same clearance package, however.
To meet requirements of the Act, one of the sponsoring of-
fice's most important responsibilities in virtually all
Contract surveys is to prepare the paperwork to obtain OMB
Clearance for any applicable information collections. This
means submitting a clearance package'' through cEannels
-80-
-------
and answering the reviewers' questions during the clearance
process.
To obtain an OMB clearance --
(1) Contact your office's Information Management Coordina-
tor or the Information Management Branch (1MB) in the
Office of Policy, Planning and Evaluation for instruc-
tions in preparing the clearance. They will help you
expedite the clearance and advise you of its progress.
(2) Complete Parts I and III of Standard Form 83 (SF 83),
"Request for OMB Review" (see Exhibit 7). Use the
instructions OMB has prepared for completing this form.
(See the Information Management Coordinator in your
office for copies.)
(3) Prepare the required supporting statement detailing and
justifying the Agency's need for the information. You
may ask the contractor to draft this statement, but re-
view it carefully to be sure it meets all OMB require-
ments. (Contact the Information Branch or your Infor-
mation Management Coordinator for help in preparing
the supporting statement and the survey questionnaire.)
(4) Submit three copies of the completed clearance form
(SF 83), the supporting statement, and the question-
naire -- collectively referred to as "the clearance
package" -- to your Information Management Coordinator
for a preliminary review for accuracy and completeness.
(5) Forward the package to OPPE's Information Management
Branch (1MB) for review and submittal to OMB.
1MB recommends that you submit the clearance package to
them (in draft or final form) for an advance review as
early as possible, preferably before the survey design is
completed. 1MB may return it with suggested changes be-
fore sending it through Agency channels to OMB.
OMB clearance can be a lengthy process, so allow ample
time for both internal and external reviews of the clear~
ance package. Allow at least two-weeks for each office
that must review the package. Under the Act, OMB has 60
days to review the package (90 days if it grants itself an
extension), but it is prudent to allow for the possibility
of a lengthy delay. A contingency plan should always be
built into the contract to avoid any legal problems should
OMB seriously delay the project or disapprove the survey
altogether.
-81-
-------
Standard Form OJ
(Rev Septembe' 1083)
Request for 0MB Review
Important
Read instructions before completing form. Do not use the same SF 83
to request both an Executive Order 12291 review and approval under
the Paperwork Reduction Act.
Answer all questions in Part I. If th:s request is for review under E.O.
12291, complete Part II and sign the regulatory certification. If this
request is for approval under the Paperwork Reduction Act and 5 CFR
1320, skip Part II. complete Pa-t III and sign the paperwork certification.
Send three copies of this form, the material to be reviewed, and k--
paperworkthree copies of the supporting statement to
Office of Information and Regulatory Affairs
Office of Management and Budget
Attention. Docket Library, Room 3201
Washington, DC 20503
PART I.Complete This Part for All Requests.
1. Departnent/agency art) Bureau/o!fice originat ng recuest
2. Agency code
3. Name of pe'son who can best answer quest'ons regarding this request
Te.eonone njrrber
( )
4. Title of nfor nation collection or rulemaking
5. Legal authority for rtornation collection or rule (cite United States Code. Public Law. or Executive Order)
USC or
6. Affected public (check all that appiy)
1 HI Indivcuals or households
2 ED State o' oca! governments
3 0 Farms
4 ~ Businesses or other *c--pro'
5 I 1 FeCeral agencies or employees
6 LJ Ncn-pro'it nstitjtions
7 Q Smali businesses 0' organizations
PART II.Complete This Part Only if the Request Is for OMB Review Under Executive Order 12291
7. Regu ation Identifier Number (RIN)
or. None assigned H]
8. Type of submission (check one in each category)
Type of review requested
Classification
Stage oftIt velopment
1 0 Standard
1 CD Maior
1 C Proposed or draft
2 G Pending
2 1 1 Nonmajor
2 1 i Final or interim final with prior proposal
3 1 i Emergency
3 1 1 Final or mter rn final, without pnor proposa
4 I ; Statutory or judicial deadline
9. CFR section affected
CFR __
10. Does this regulation contain reporting or recordkeeping requirements that require OVB approval under tie Paperwcrn Reduction Act
and 5CFR 1320?
~ Yes
: Nc
11. If a major rule, is the-e a regulato-y impact analysis attached1
If No, ' did QiV.B waive the analysis?
1 ~ Yes 2
3 C Yes 4
: j Nc
Gnc
Certification for Regulatory Submissions
In submitting tnis request for OMB 'eview, the authorized regulatory contact and the p'og'am oHic a certfy that the requirements o' E 0. 122S1 and any appl
po icy directives have Seen complied with.
Signature of program official
Date
Signature o< authorized regulate-/ contact
Date
I
12. (OMB use only)
Previous edrt-ons obsolete 83 108 Standard Form 83 -t Hi'
NSN 7540-00-634-4034 Pt#sr:*i: t. OUS
3CrR .3ro.<-" 1 c :rri;
-------
EXHIBIT 7
PART III.Complete This Part Only II the Request is for Approval ol a Collection
of Information Under the Paperwork Reduction Act and 5 CFR 1320.
13. AbstractDescr be needs, jses and affected public in 5C >voras or less
14. '/pe of in'orTat'On co lection (check only one)
Information collectioni not contained in rufat
1 CH Regular submissior
Information collections contained In rules
3 ID Existrg regulation (no change prooosed)
4 EI Notice of proposed 'jlemakmg (N"RM)
5 ~ Fnal, NPRM was prev cjsly published
2 ID Emergency sjbm.ss on (certification attached)
6 Final or inte'im 'inal withojt prior NPRM
aG Regjlar sjSiti ss on
B \_J Emergency submssion (cert'ficatior. attached)
7. Erter date of expected or actual Federal
Register pjol caticn a; this stage a' rulema* rg
(month, day, year):
15. Type cf 'eview requested (check cnlyor.e)
1 [D New collection
2 | Revision of a cjrrently approved collection
3 I Extens on of the expration date of a cur-ently approved collection
without any charge in the substance 0' n the method o' co lectior
4 ~ Reinstatement of a previously approved collection for which app-oval
has expired
5 CD Existing collection in use without an OM3 control njrrber
16. Agency report to-m ru.T.be'(s) (include standard/optional form ruiiber(s))
17. Annua resort rg or disclosjre burden
1 Number cf -espondents
2 Number of 'esponses per respcrdent . .
3 Tota annual responses (line J times line 2)
4 He jrs per response
5 Total houriyCr/He 3 times iine 4) . . .
18. AnnLa recordkeeping burden
1 Number of recarCkeepers
2 Annual hoLrs pe-recorckeeper. .
3 Tota: recc'dkeeping hours(7'r?e ฆ! times t're 2)
4 3ecordess line 2)
Explanation of dlfferenct
4 s'ogram change
5 Adiustme.nt . .
23. Frequency of recordkeeping or reporting (cnecx a!l that apciy)
1 CD Recordkeeping
Reporting
2 ID Or occas on
3 ~ Weekly
4 CD Monthly
5 CD Quateriy
6 CD Semi- annjally
7 CD Annually
8 CD B ennially
9 CD Other (describe)'
20. Cj"rent (most recent) CMB corfol .number or comment number
21. Requested expirat.cn date
24. Respondents' obligation to COT,ply fc.'iecfc the strongest obligation tnatapplies)
1 ID Voluntary
2 ~ 3eq jired to obtain or retain a benefit
3 CD Mandatory
25. A-e the respordents primari'y educational agencies or institutions or is the primary purpose of tne collection related to Federal education programs? CD Yes CD No
26. Does the agency use sampling to select respondents o' does the agency recommend or prescribe the use of sampling or statistical analysis
3y respondents? . . .. ... ... ... ... ... ...
CD Yes CD No
27. Regulatory author.ty for the irfo'mation collection
CFR
FR .
; or. Other (specify);
Paperwork Certification
In subr-vtting this request for OMB app'oval the agency head, the senior official or an authorized representative, certifies that the requirements of 5 CFR 1320, the
Privacy Act. statist cal standards 0'd red ves, and any other aoplicable info'matior poi.cy directives nave been complied w th
Sigrature o' program off'Cial
Date
Sigrature of agency head tre sen or off c a' o' an autnonzeo representative
Date
-------
Keep Che contractor regularly advised of the status of
the clearance. In no case should the contractor contact
1MB, other Agency officials, or OMB directly regarding the
status of the package.
When OMB grants the clearance, they will assign a control
number for the information collection activity. Both the
control number and the expiration date must appear in spe-
cific locations on the questionnaire and other interview-
ing instruments. Be sure to give the contractor clear in-
structions about OMB's requirements in this regard.
After the survey is in progress, send the Information Man-
agement Branch of OPPE copies of the final printed ques-
tionnaire(s) , the letter of transmittal and instructions
to the respondents, and other relevant written materials
EPA or the contractor sends to the respondents during the
course of the survey. 1MB will retain the copies in its
official clearance files.
If, at any point in the data collection process, either
you or the contractor decides to make any significant
change in the questionnaire or the methods used to collect
the information, you must obtain formal approval from OMB.
Similarly, if the data collection is not completed by the
expiration date of the clearance, you must apply for an
extension. Consult the Information Management Branch or
your Coordinator about the procedures for obtaining these
additional OMB approvals.
C. MONITORING THE CONTRACTOR'S PERFORMANCE
This section (1) examines the three most common mechanisms
for monitoring the contractor's performance during the sur-
vey and (2) provides tips on controlling the quality of the
data collection and data processing activities.
1 . Monitoring Mechanisms
Once the contract is awarded, the project officer's
relationship with the contractor takes on an operation-
al dependency. This means exercising critical judgment
in determining how much oversight of the contractor's
work is necessary and where it should be focused. Dif-
ferent surveys, depending on their size and complexity,
require different degrees of oversight.
The principal mechanisms available to the sponsoring
office for overseeing the contractor's adherence to
the schedule, budget, and technical provisions of the
-84-
-------
contract are (a) contractor progress reports, (b) oral
briefings or presentations, and (c) on-site visits to
the contractor's facility and the interviewing sites.
These mechanisms are examined below.
Progress reports.
Periodic progress reports from the contractor at
intervals stipulated in the contract normally pro-
vide most of the information you will need to moni-
tor the status of the survey. These reports should
summarize:
Progress in completing required tasks and deliv-
erables ;
Work planned for the next scheduled reporting
period;
Any problems that nay affect the scheduled tasks
or deliverables;
Any preliminary technical findings (even though
they are tentative and not intended for wide
distribution); and
The cost status of the contract.
The Agency's standard monthly progress report clause
for cost-reimbursable contracts requires the con-
tractor to track the financial status of the con-
tract. This will keep you up to date on the rela-
tionship of expenditures to progress under the con-
tract. We recommend that you ask the contractor to
graph the cost status of the contract for you. This
is an effective way of preventing cost overruns --
more effective than the expense vouchers contractors
periodically submit for completed work. Expense
vouchers do not give an up-to-date record of actual
expenditures because frequently there is a timelag
of several months before the contractor bills the
Agency for completed work. (See Exhibit 8 for an
example of a well-prepared cost status report.)
Briefings or presentations.
Periodic oral briefings or presentations by the con-
tractor at various milestones -- the completion of
the pretest or the data processing operations, for
example -- are also effective means of tracking the
contractor's performance. These may be held at the
-85-
-------
contractor's facility or at EPA. To minimize travel
costs, telephone briefings may sometimes suffice,
particularly for small-scale surveys.
On-site inspections.
Effective monitoring tools include: personal visits
to the contractor's main facility, the pretest site,
and one or more field locations when the interview-
ing is in progress. On-site inspections are an ex-
cellent way of checking on the contractor's adher-
ence to the quality-control procedures in the con-
tract.
For a major survey, we recommend that you use all of
the above monitoring mechanisms to ensure that the con-
tractor is performing work in accordance with the terras
of the contract.
2. Monitoring the Data Collection and Data Processing
Stringent quality control by the contractor at each
step of the (a) data collection and (b) data process-
ing is necessary to ensure that errors and biases are
kept to a minimum.
Below are some suggestions for monitoring the contrac-
tor's performance during these two major phases of the
survey.
Quality control of the data collection activities.
During the data collection phase, carefully check
the contractor's performance in executing four steps
in particular: (1) the pretest, (2) the training of
staff personnel, (3) the follow-up of interviews,
and (4) the data validation and verification.
(1) The pretest.
As pointed out in earlier chapters, the most
effective way of ensuring the success of the
data collection is thorough a pretest of the
questionnaire and, if feasible, of the proposed
field procedures as well (a pilot test).
To help guarantee the validity of the pretest,
have a survey statistician review the contrac-
tor's plan and the draft questionnaire before
the pretest begins. Afterwards, make sure any
modifications suggested by the pretest results
are incorporated into the survey proper.
-86-
-------
EXHIBIT 8
EXAMPLE OF A COST STATUS REPORT
% $
Expended
Expended
100-"
Projected
Actual
353,613.60
90--
80--
314,323.20
70--
275,032.80
50"
157,161.60
40--
117,871.20
78,580.80
39,290.40
10--
May June July Aug Sept Oct Nov Dec Jan Feb Mar Apr May
1982 1983
-87-
-------
(2) Training of contractor personnel.
Proper training of the interviewers and other
personnel involved in the data collection oper-
ations is also critical to the success of this
phase of the survey.
If the contractor is collecting the data through
telephone or personal interviews, it is espe-
cially important to have a survey specialist re-
view the contractor's plans for recruiting and
training the interviewers. Then, before the
training and field activities get under way,have
the specialist check the contractor's training
guidelines and handbooks for the interviewers
and the people who will do the editing, coding,
and quality control operations. These materials
should be scrutinized for consistency and adher-
ence to the procedures in the contract.
In addition, for a major survey, you or a key
member of your staff should attend at least one
of the interviewer training conferences to ex-
plain the Agency's research objectives and poli-
cies regarding contact with the public, and to
resolve any substantive issues concerning the
interviewing procedures before the field opera-
tions begin.
(3) Follow-up of the interviews.
After the interviewing is in progress, you will
want to make occasional visits to the contrac-
tor's main facility (and the field locations,
if appropriate) to make sure the contractor is
adhering to the procedures specified for follow-
ing up the initial interviews.
For example, the contractor should make no sub-
stitutions of respondents (unless specified in
the contract), and should repeat interviews (or
mailings) on a sample of the respondents and
some nonrespondents to make sure there has been
no bias against individual interviewers.
(4) Data validation and verification.
When the interviewing of the initial sample is
complete, the contractor should provide you
with a detailed accounting of the statistical
results. Carefully review these preliminary
-88-
-------
findings to make sure the contractor has at-
tained the target response rates, weighted the
data and made adjustments for nonresponses.
Later, when the data collection operations are
complete, review all quantitative reports on
the outcome of the validation and verification
of the interviews. We suggest that you also
have a statistical expert review these reports.
Quality control of the data processing and analysis
activities.
Overseeing the contractor's (1) data processing ac-
tivities and (2) analytic work, if applicable, are
equally important.
(1) Data processing control.
Site visits are important to ensure that the
contractor rigorously controls the data tabula-
tion operations.
The purpose of these visits is to make sure the
contractor is using proper procedures to mini-
mize human error and control bias in the coding,
key punching, and transcription operations. For
example, as the questionnaires are returned or
the interviewing completed, the questionnaires
and interviewer notes should be checked before
they are processed and coded.
Open-ended questions should receive special han-
dling and coding. You may want to have members
of your staff work with the contractor in devel-
oping codes for open-ended questions and proper-
procedures to deal with missing or "unanswered"
quest ions.
Also check the editing of the data tapes. Once
data have been entered into the computer, the
contractor should edit the tapes to eliminate
or clarify inconsistent or illogical entries.
(See Chapter 6 of Volume II for details of the
data processing activities.)
(2) Data analysis.
The final result of the data processing opera-
tions is a "clean," virtually error-free data
-89-
-------
file. This serves as the basis for the statis-
tical analysis that will answer the concerns the
survey was designed to address.
Proper management of the data analysis is vital.
The type of oversight the sponsoring office gives
to the analysis depends largely on whether the
Agency or the contractor compiles the statis-
tics.
If the Agency is required to prepare the analy-
sis, make sure that the data file includes what-
ever information the project staff will need to
calculate sampling error. If the contractor is
required to do the analysis, have a statistical
expert review the contractor's estimates of sam-
pling error before the analytic work is started.
Finally, regardless of who prepares the analy-
sis, have a statistical expert review both the
analysis techniques and the findings for accu-
racy, consistency, and completeness.
(See Chapter 7 of Volume II for details of the
analysis process.)
D. PROVIDING ADMINISTRATIVE SUPPORT
In addition to having total responsibility for technical
oversight of the contract, the project officer also must
assist the contracting officer in fulfilling certain ad-
ministrative provisions.
These support activities include (1) maintaining records
of the survey, (2) reporting contractor progress, (3)
authorizing contractor payments, (4) recommending approval
of subcontracts, (5) evaluating personnel changes, (6)
guarding against cost overruns, (7) recommending approval
of contract modifications, (8) evaluating contractor per-
formance, and (9) helping to close out the contract.
The specific tasks for which you are responsible are de-
scribed briefly below.
1. Maintaining Records of the Survey
You must maintain complete, up-to-date,accurate records
of the survey. Thorough recordkeeping and well-main-
tained files are important in case key project staff
leave before the contract is closed out.
-90-
-------
Be especially careful to keep records concerning the
contractor's adherence to the terms of the contract
and the schedule. You will need this information to
evaluate the contractor's performance. Also retain
copies of all deliverables so you can answer inquiries
after contract close-out.
2. Reporting Contractor Progress
Keep all interested and involved parties -- your con-
tracting officer, your supervisors, other agencies,
advisory groups,etc. -- informed about the contractor1s
progress.
Be sure to report any deviations from the terms of the
contract to your contracting officer, such as
Any performance failure by the contractor;
ฎ Any proposed changes in key staff;
ฎ Delays in the schedule;
ฎ Lapses in technical requirements;
Any changes in the scope of the contract;
Inappropriately incurred costs;
e Deviations in travel expenditures;
ฎ Misuse of patented items or privacy data; and
Any failure by the Agency to meet its contractual
obligations.
While you should assist the contractor in any way that
will further the Agency's interests, be careful not to
do anything that may affect the scope or direction of
the contract without obtaining prior approval from the
contracting officer.
3 . Authorizing Contractor Payments
To control contractor expenditures and ensure that
EPA meets its research objectives, provisions in cost-
reimbursable contracts require prior authorization and
approval by the contracting officer for --
Subcontracting activities (with some exceptions);
ฉ Acquisition of equipment;
-91 -
-------
Reimbursement of costs exceeding the limitations
established in the contract; and
Reimbursement for overtime or premium wage payments.
Be careful not to give the contractor permission to
spend contract funds for any of the above without the
contracting officer's approval. However, because all
these activities relate to the technical provisions of
i the contract, your contracting officer generally will
~ consult you before withholding or granting authority
for payment of these expeditures.
4. Recommending Approval of Subcontracts
EPA has no authority to direct the performance of sub-
contractors but must work through the prime contractor.
As indicated above, the contractor must obtain prior
Agency approval for any subcontracting arrangement.
The contracting officer will ask you for a recommenda-
tion before approving any such arrangement.
5. Evaluating Personnel Changes
The contractor must notify the Agency reasonably well
in advance if it becomes necessary to replace key per-
sonnel after the contract is awarded. Any substitution
requires prior written consent from the Agency. The
contracting officer also will ask you for your recom-
mendation before approving any substitution request.
6. Guarding Against Cost Overruns
Cost overruns occur when a contractor exceeds the nego-
tiated estimate on a cost-reimbursable contract. (Gen-
erally, cost overruns do not apply in a fixed-price
contract because the contractor is legally responsible
for any costs exceeding the total price.)
Contractors are required to notify the Agency whenever
the total cost of the survey may be higher or signifi-
cantly lower than the negotiated amount. By keeping
track of the expenditures in progress reports and other
monitoring mechanisms (discussed earlier in this chap-
ter) , you may be able to anticipate a potential cost
overrun.
However, if at any point in the survey you have reason
to believe that a cost overrun is likely, bring it to
the attention of the contracting officer immediately
and recommend that the Agency --
-92-
-------
-- Terminate the contract before the overrun occurs;
-- Decrease the amount of work of the contract so no
additional funds are needed; or
-- Increase the contract funds so the contractor can
carry out the additional work.
If you recommend that additional funds be made avail-
able and the contracting officer agrees, prepare a new
procurement request and rationale statement and forward
them through channels to the Procurement and Contracts
Management Division (PCMD). (See Chapter 4 for proper
procedures.)
7. Recommending Contract Modifications
Sometimes it is necessary to make changes in the scope
of work or to include additional coverage for situa-
tions not anticipated during the negotiations. How-
ever , you must not authorize the contractor to modify
the scope of work in any way. Only the contracting
officer can 3o that. Any unauthorized modification
makes the Agency vulnerable to a claim for the addi-
tional costs the contractor may incur.
The contracting officer generally has the authority to
direct the contractor to make changes in the scope of
wor"k provided an equitable adjustment is made in the
price and/or delivery schedule. All contract modifi-
cations must be formalized in a written agreement be-
tween the contracting office and the contractor.
8 Evaluating Contractor Performance
You are responsible for evaluating the deliverables as
they are submitted and for making recommendations to
the contracting officer regarding their acceptability.
Advise the contracting officer if you believe that con-
tractor payments should be withheld because certain
terms of the contract have not been met. Both fees and
progress payments can be entirely or partially withheld
pending resolution of the problem.
9. Helping to Close Out the Contract
The final step in the administration of the contract
is referred to as "contract close-out." This is a
formalized procedure triggered by the submission of all
required deliverables and/or the completion of all the
specified tasks.
-93-
-------
As soon as the contractor has completed all work on
the contract to your satisfaction, notify the contrac-
ting officer in writing so the contract can be closed
out quickly and efficiently. You also must complete
an EPA form evaluating the contractor's technical per-
formance, and submit all records pertaining to the
administration of the contract to the contracting
officer (except material that would compromise the
confidentiality of the respondents).
In this chapter we examined the project officer's
principal post-award responsibilities: monitoring,
controlling, and evaluating the contractor's technical
performance; maintaining the Agency's technical commit-
ments and supporting the administration of the contract.
In the next chapter we discuss the final phase of the
survey -- the documentation and release of the findings.
-94-
-------
CHAPTER 8
REPORTING AND DISSEMINATING THE FINDINGS
Virtually all survey contracts require a comprehensive final
report. For a major survey, more than one final report may have
to be written to communicate the data and statistics effectively
to both technical and general audiences.
This chapter examines the project officer's role in --
Overseeing the preparation of the contrac-
tor's final report;
Circulating drafts for review and comment;
and
Releasing and disseminating the survey
findings.
A. OVERSEEING PREPARATION OF THE FINAL REPORT
As project officer, one of your most important responsi-
bilities during the final phase of the survey is to over-
see the documentation of the survey. In most cases, this
involves reviewing the outline and each draft submitted
by the contractor.
All survey reports should be issued as EPA documents.
Therefore, it is essential that you carefully review all
draft reports submitted by the contractor for accuracy,
completeness, and appropriateness for the intended audi-
ence. Do not neglect early drafts because conceptual and
structural errors are difficult (and expensive) to correct
in the final stages of the writing.
The survey findings should not be overstated or sensation-
alized. Statistical information concerning health risks or
health consequences, in particular, should be presented in
a way that will not unduly alarm affected subpopulations,
businesses, local communities, etc.
While it is beyond the scope of this handbook to give de-
tailed instructions on how to write a survey report, below
are some general suggestions for (1) focusing and (2)
-95-
-------
organizing a final survey report. In addition, Exhibit 9
presents an outline for the final report. We suggest that
you pass these ideas along to the contractor or, in the
event the Agency is required to write the final report, to
your staff for guidance.
1. Focusing the Content
What is said and hov it is said depends on the intended
audience --
If the primary audience of the report is engineers
and scientists^ a detailed account of both the meth-
ods the researchers used to obtain the data and the
analytical findings is appropriate. Extensive sup-
port documentation should be included as well, e.g.,
the questionnaire, the survey design specifications,
interviewer training materials, the coding and keying
instructions, the data collection and data processing
procedures, and the tabulation and analysis tech-
niques.
On the other hand, if the report is intended for a
general audience, the emphasis should be on the re-
sults of the research -- what was learned rather than
how the information was obtained. The text should
be written in plain English and contain a minimun
of technical and statistical terminology.
2. Organizing the Text
The survey report should include (1) an executive sum-
mary, (2) an introduction describing the background and
objectives of the research, (3) a description of the
methodology, (A) the data tabulations and analytic find-
ings, (5) the major conclusions, and if appropriate, (6)
recommendations for further study or action. More spec-
ifically --
(1) Executive summary.
An executive summary is essential for any survey
report more than ten pages long. The summary should
be placed at the beginning of the report. In one
to four pages it should discuss the purpose and
significance of the study, the major technical find-
ings and conclusions, and the potential applications
of the research. Any comments on how the informa-
tion was obtained should be kept to a minimum. The
focus should be on the results -- what was learned.
-96-
-------
EXHIBIT 9
SUGGESTED CONTENT OF A SURVEY REPORT
EXECUTIVE SUMMARY (OR ABSTRACT)
0 Purpose of the survey.
Brief summary of the results or conclusions.
Summary statement of the significance of the research
and its possible applications.
INTRODUCTION
Statement of the research objective (why the survey was
undertaken), its relation to the Agency's Mission, and
the legal authority under which the study was conducted.
ฎ Summary of previous research and publications on the
problem.
Major survey design features (target population,
coverage, sampling error, response rate, etc.).
General approach used to accomplish research objectives.
0 Explanation of any unusual or original features of the
survey.
Statement of the organization of the main body of the
report.
METHODOLOGY
Description of the sample design (sampling frame, sample
size, selection procedures, estimation procedures, pro-
cedures for estimating variances, etc.).
Procedures used to pretest the questionnaire and field
operations.
-97-
-------
EXHIBIT 9
Continued
Data collection procedures (procedures used to select,
train, and supervise interviewers and validate their
work; methods used to achieve the target response rate;
methods and techniques used to minimize or control
response errors; folTow-up techniques; etc.).
Data processing procedures (procedures used to super-
vise and validate coding, keypunching, and transcription
operations).
Any other quality control methods.
Estimates of sampling errors and problems with the data
such as missing values, biases, low response rates, etc.
Tabulation and analysis procedures.
9 Procedures for protecting the rights of data subjects
and respondents, and for safeguarding confidentiality.
SURVEY FINDINGS
The presentation of the findings is at the discretion of
the writer. Be careful to separate the descriptive find-
ings from any interpretive or analytic findings.
CONCLUSIONS
Summary of the findings.
Agency assessment of the findings and limitations of the
data.
9 Summary of the contributions of the survey (what was
accompli shed) .
Discussion of the significance of the findings and
potential applications.
RECOMMENDATIONS (If appropriate)
-98-
-------
(2) Introduction.
This part of the report should summarize the objec-
tives of the research, previous research on the sub-
ject, the major survey design features, the general
approaches used to accomplish the research objec-
tives, and any unusual features of the study. It
also should include a brief description of how the
remainder of the report is organized. The intro-
duction should not include data tabulations, analy-
tical findings, or conclusions.
(3) Methodology.
This part should discuss how the researchers obtained
the information for the main body of the report --
the the survey findings (see part 4 below).
If the report is intended for a general audience,
methodological details should be kept to a minimum.
In fact, it often is best to summarize the method-
ology in the introduction or an appendix rather than
present it in a separate section.
For a technical audience, however, a chapter on
methodology is a must. It should include descrip-
tions of the techniques used to develop the ques-
tionnaire, the sampling plan, and the response
rates; the procedures used to select and train in-
terviewers; the methods used to collect, process,
and weight the data; and any other "process" de-
tails. It is essential to include information on
sampling error.
(4) Survey findings.
This is the main body of the report, which should
contain the descriptive and interpretive (analyti-
cal) findings. It may consist of several chapters.
However the findings are organized, the descriptive
findings (tabulations and summaries of raw data)
should be clearly distinguished from the interpre-
tive findings (inferences or interpretations drawn
from the data). Forecasts and projections, too,
should be clearly identified.
To enhance reader understanding, line graphs, bar
graphs, and pie charts should be used to illustrate
major findings, especially for a nontechnical audi-
audience. Graphics should not substitute for text,
-99-
-------
however. They should highlight and clarify the
text.
Charts, graphs, and tables should be clearly labeled
with a title, the time period to which the data
refer, the affected geographic areas, and the units
on the vertical and horizontal scales. Whenever
possible,they should be placed immediately following
the page on which they are referenced. Tables on
top of tables should be avoided; series of tables
should be placed in an appendix.
(5) Conclusions.
This part of the report should summarize the major
findings. Whether a contractor or the project staff
has written the report, you must include EPA's as-
sessment of the quality of the statistics, any con-
ceptual or other limitations of the data, the sig-
nificance of the research, and the potential appli-
cations of the findings. It is also a good idea
to compare the findings with research results of
other studies similar in size and scope.
(6) Recommendations.
This part is optional. If used, it should include
the researchers' recommendations for additional
study or action to further the Agency's initial
purpose in undertaking the survey.
B. CIRCULATING DRAFTS FOR REVIEW AND COMMENT
When the final draft is completed, submit it for review
and comment, as appropriate, to --
Other technical experts inside and outside the Agency;
= An Agency statistical expert;
= Any relevant advisory committees;
Other Federal agencies that have contributed to the
study;
State and local officials in affected or participating
jurisdictions; and
Your Office Director and other EPA officials.
-100-
-------
It is especially important to have a statistical expert
review descriptions of the survey methodology and comment
on the statistical statements and conclusions -- e.g.,
whether the analysis has taken proper account of weights,
sampling errors and limitations of the data. This expert
also can help you write the Agency's assessment of the
statistics, an important part of the "conclusions" section
of the
report.
For a major survey, you may want to circulate one or more
preliminary drafts of the report to these same people
durng the course of the survey. It may avoid extensive,
costly, last-minute revisions.
C. RELEASING AND DISSEMINATING SURVEY DATA TO THE PUBLIC
This section describes Agency procedures for (1) releasing
and disseminating survey data, (2) reviewing and approving
contract survey reports, (3) issuing press releases, and
(4) holding press conferences.
1. Procedures for Releasing Survey Data
The Agency has no specific policies governing the re-
lease of survey data to the public. However, to pre-
vent the premature release of statistical findings
during the course of the survey and ensure that the
results of Agency-sponsored research are reported ac-
curately and with appropriate sensitivity to the im-
pact they may have on affected populations, organiza-
tions, businesses, or communities,we recommend that
(1) All reports concerning an Agency-sponsored survey,
whether oral or written, be issued by EPA, not the
contractor.
(2) The Director of the Office sponsoring the survey
(or a higher-level EPA official) should give writ-
ten approval for the release of new survey data or
analytical findings before they are made public.
(3) If other agencies have participated in the infor-
mation collection activity, appropriate officials
of those agencies should be notified before the
data are released. It also may be advisable to
brief State and local officials or community lead-
ers of the affected locales before releasing impor-
tant or "sensitive" findings.
-101-
-------
(4) All survey reports should be disseminated by the
Agency.
(5) Agency staff should initiate any press notices or
press conferences announcing the release of new
survey data or findings. A staff statistician
knowledgeable in the subject area should be present
at any meetings with the press when survey results
are announced.
The above recommendations generally refer to the ini-
tial release of survey data. Once the Agency makes the
information public, the contractor is usually allowed
to publish or otherwise use the data after a specified
date, provided the confidentiality of the respondents
is maintained. Data rights and any relevant limita-
tions should be included in the survey contract.
2. Issuing Press Notices
We recommend that the sponsoring office issue a press
release before announcing sensitive findings or issuing
a survey report of major impact.
To initiate a press release, submit a draft of your
announcement to the press liaison in your office. In-
clude a summary of the background and significance of
the study as well as the data or findings you want to
make public. Include the names and phone numbers of
staff to contact for further information. The press
liaison will edit the text and submit it to your Office
Director for approval before sending out the release.
3. Holding Press Conferences
When you intend to release survey data of major public
interest, consider holding a press conference to ex-
P1 ain important technical features of the study, to
reiterate the Agency's position on the issues, and to
answer questions from the press. Your office's press
liaison will arrange for the conference. The sponsor-
ing office, however, is responsible for making sure a
knowledgeable statistician is present at the press
conference.
YJe recommend that you coordinate any press conferences
with officials of jurisdictions likely to be most af-
fected by the findings and invite them to participate.
-102-
-------
In this final chapter of Volume I of the Survey Man-
agement Handbook, we discussed the project officer's
principal responsibilities during the last phase of
the survey -- reviewing draft reports submitted by the
contractor, circulating them for comment or approval,
and releasing and disseminating all reports about the
survey.
Remember to consult the "Help Directory" located at
the beginning of this volume if you need more infor-
mation on designing the survey, preparing a schedule
and budget, initiating and managing the procurement, or
obtaining OMB clearances.
Volume II of the handbook presents additional informa-
tion on managing surveys after a contractor is hired.
It is organized around the survey work plan, in which
the contractor describes the methods and procedures
to be used in collecting, processing, and analyzing
the data from or about the survey population. The
principal components of the work plan normally are:
an analysis plan, specification of the method(s) of
collection, a draft questionnaire and specifications
for any pretests, a sampling plan, the interviewing
procedures, and the data processing procedures. Vol-
ume II also includes a glossary of terms and a list
of recommended sources.
-103-
-------
ACROYNMS AND ABBREVIATIONS
CBD Commerce Business Daily
EPA U.S. Environmental Protection Agency
JNCP Justification for Noncompetitive Procurement
OMB U.S. Office of Management and Budget
PASS Procurement Automated Source System
PCBs Polychlorinated biphenyls
PRA Paperwork Reduction Act
RFP Request for Proposal
RIA Regulatory Impact Analysis
SMSA Standard Metropolitan Statistical Area
EPA OFFICES
1MB Information Management Branch, Regulation and
Information Management Division, OSR
MIDSD Management Information and Data System Division
OARM Office of Administration and Resources Management
OECM Office of Enforcement and Compliance Monitoring
OGC Office of General Counsel
OIRM Office of Information Resources Management
OPPE Office of Policy, Planning, and Evaluation (formerly
Office of Policy and Resource Management)
OSR Office of Standards and Regulations, OPPE
PCMD Procurement and Contracts Management Division, OA
RIMD Regulation and Information Management Division, OSR
SPB Statistical Policy Branch, Chemicals and Statistical
Policy Division, OSR
-105-
-------
APPENDICES
-------
APPENDIX A
Page 1 of 3
EXCERPTS FROM AN OMB CLEARANCE PACKAGE
PREPARED FOR EPA BY RESEARCH TRIANGLE INSTITUTE
AUGUST 17, 1982
From "Study on Carbon Monoxide Exposure of Residents of
Washington, D.C., and Denver, Colorado"
Program Objectives and Uses of Data
This study is designed to provide a systematic measurement of
actual exposures of individuals to CO [carbon monoxide] . 1..a
distribution of individuals to be studied is constructed so as
to provide a representative range of exposures throughout an
individual's 24hour period. The temporal distribution of ex-
posures is to be recorded based upon activity pattern informa-
tion from all the participants in the study.
The immediate aim is to validate a methodology for determining
population CO exposures in two urban areas (Washington, D. C,
and Denver, Colorado) during the period of maximum ambient CO
concentrations (October through December). The population
exposure profile will be determined by direct measurement of CO
with personal exposure monitors (PEM's) through the use of sta-
tistical inference from the statistically drawn sample. The
study intends to provide sufficient data to determine exposure
as a function of concentrations -"ithin significant microenviron-
ments (home, in-transit, work and leisure) and individual acti-
vity patterns. This will be contrasted to exposure estimates
obtained from fixed monitoring results. The study attempts to
evaluate the degree to which fixed site monitors may be utilized
for estimation of population exposures.
Some specific objectives include the following:
To develop a methodology for measuring the distribution of
carbon monoxide (CO) exposures of a representative popula-
tion of an urban area for assessment of the risk to the
population. (It should be noted that this development and
evaluation must occur prior to revisions to the CO Criteria
Document scheduled for about 1986,...)
To test, evaluate, and validate this methodology by em-
ploying it in the execution of pilot field studies in
Denver, Colorado, and in Washington, D. C.
To collect personal exposure data of sufficient quality
to permit the Environmental Protection Agency to compare
it with fixed-site data in order to (a) evaluate the rep-
resentativeness of official air monitoring stations for
-------
APPENDIX A
Page 2 of 3
estimation of population exposure, and (b) evaluate the
feasibility of improving the existing CO siting criteria
to better represent air pollution exposures of the popu-
la t ion.
To obtain an activity-pattern data base related to CO
exposures. This data base will permit the validation of
two developed but untested CO exposure models in use with
in the Agency (SHAPE and NEM)....
Authori ty
The authority for conducting the proposea research is found in
the Clean Air Act, Title 42 U.S.C. 7410, 7619.
Section 110(c) requires that the states have programs for es-
tablishing and operating methods to monitor, sample, and ana-
lyze data on ambient air quality. Section 319 requires promul
gation of regulations establishing uniform air quality monitor
ing criteria and methodology for measuring air quality accord-
ing to a uniform air quality index and provides for uniform
air quality monitoring systems in major urban areas and other
appropriate areas. In order to establish uniform criteria for
the operation of these air quality monitoring systems and to
develop and evaluate criteria for the siting of fixed monitor-
ing stations that form the heart of these systems, it is essen
tial to obtain field data on exposure frequency distributions
of the population of major urban areas operating monitoring
systems in compliance with Section 110(c) and 319 of the Act.
In addition, Section 104 requires the Adminstrator to set a
standard with an ample margin of safety to protect the human
health. To determine whether the public health is being pro-
tected at the desired margin of safety, it is necessary to de-
termine the level to which the public is exposed. This infor-
mation is reviewed every five years and published in a revised
criteria document for the pollutant....
Target Population
The target population for the carbon monoxide exposure study
includes all non-institutionalized, non-smoking residents of
the urbanized portion of the Washington, D. C. SMSA and the
Denver, Colorado metropolitan area who are at least 18 years
of age at the time of the study. The estimated population siz
for these groups is 1-1.2 million in Washington, D. C. and
245,000 in Denver, Colorado...
These sites were chosen because (1) they differ in elevation
and CO levels, (2) of the interest expressed by officials in
-------
APPENDIX A
Page 3 of 3
Denver, and (3) of the extensive commuting activities in Wash-
ington, D. C. A selection of 1,250 individuals is proposed
in order to complete 1,000 interviews in Washington, D. C. A
sample of 625 individuals is proposed for Denver, and it is
intended to produce 500 completed interviews for the first
season (October - December). The 500 respondents will eaph
collect CO monitoring data for two days. An optional second
season in Denver (summer) is intended to produce approximately
125 completed interviews so that a two season Denver study
would produce 625 completed interviews....
-------
APPENDIX B
Page 1 of 12
EXAMPLE OF AN EPA SURVEY QUESTIONNAIRE
STUDY OF CARBON MONOXIDE EXPOSURE
Sponsored by:
Environmental Protection Agency
Washington, DC 20460
Conducted by:
Research Triangle Institute
P.O. Box 12194
Research Triangle Park, NC 27709
PARTICIPANT QUESTIONNAIRE
RESEARCH TRIANGLE INSTITUTE IS UNDERTAKING A RESEARCH STUDY FOR THE U.S.
ENVIRONMENTAL PROTECTION AGENCY TO ASSESS MAN'S EXPOSURE TO CARBON MONOXIDE.
THE INFORMATION RECORDED IN THIS QUESTIONNAIRE WILL BE HELD IN STRICT CON-
FIDENCE AND WILL BE USED SOLELY FOR RESEARCH INTO THE EFFECTS OF ENVIRONMENTAL
FACTORS ON PUBLIC HEALTH. ALL RESULTS WILL BE SUMMARIZED FOR GROUPS OF PEOPLE;
NO INFORMATION ABOUT INDIVIDUAL PERSONS WILL BE RELEASED WITHOUT THE CONSENT
OF THE INDIVIDUAL. WHILE YOU ARE NOT REQUIRED TO RESPOND, YOUR COOPERATION IS
NEEDED TO MAKE THE RESULTS OF THIS SURVEY COMPREHENSIVE, ACCURATE, AND TIMELY.
Study Number:
-------
APPENDIX B
Page 2 of 12
INTRODUCTION
The questions in this questionnaire ask about your home, your commuting trips,
your place of work, and your leisure activities. Some questions may not apply
to you; you are instructed in the questionnaire to skip those questions.
Please read the questionnaire carefully and follow all instructions. To
complete this questionnaire, you should circle a number or write in your
answer. If you have any questions about this questionnaire, the interviewer
will be happy to assist you when he or she returns to collect this question-
naire, the monitor, and the Activity Diary.
The first set of questions deals with your home or living quarters.
1. Which of the following best describes your living quarters?
CIRCLE ONE NUMBER.
01
One-family house detached
from other houses
02
One-family house attached
to one or more houses
03
Duplex
04
Apartment
05
Mobile home or trailer
06
Rented room
07
Boat, van, etc.
2. Approximately how many square feet are in your living quarters?
square feet
3. a. How many people live with you in your living quarters?
people - IF "NONE," GO TO QUESTION 4.-
b. How many people who live with you smoke tobacco in any form?
people IF "NONE," GO TO QUESTION 4.
c. How many people who live with you smoke cigarettes?
people + IF "NONE," GO TO QUESTION 4.
d. Approximately how many packs of cigarettes are smoked while you are
at home during a normal week?
CIRCLE ONE NUMBER.
01 Less than 1 pack
02 1-4 packs
03 5-7 packs
04 8 or more packs
-------
APPENDIX 3
Page 3 of 12
4. Are the following items used in your living quarters, and if so, are they
vented to the outside?
CIRCLE ONE NUMBER ON EACH LINE.
Yes, used
and vented
Yes, used but Not
not vented Used
A. Fireplace
B. Wood stove
C. Gas furnace
D. Gas cooking stove
E. Gas hot water heater ....
F. Gas clothes dryer
G. Gas or kerosene space heater
H. Other gas appliance ....
+
(SPECIFY)
2
2
2
2
2
2
2
2
3
3
3
3
3
3
3
3
5. Are the following energy-saving devices present where you live?
CIRCLE ONE NUMBER ON EACH LINE.
Yes No Don't Know
A. Storm windows 1 .... 2 3
B. Storm door(s) 1 .... 2 3
C. Extra insulation 1....2 3
D. Special dampers in stove or
fireplace 1 .... 2 3
E. Other device 1 .... 2 3
4-
(SPECIFY)
6. In your living quarters, do you ever use the following types of fans?
CIRCLE ONE NUMBER ON EACH LINE.
Yes No
A. Attic fan 1 .... 2
B. Window fan 1 .... 2
C. Other type of fan 1 .... 2
ฆJ.
(SPECIFY) '
-------
APPENDIX B
Page 4 of 12
7. What is the main type of heating system at your living quarters?
CIRCLE ONE NUMBER.
01 Steam or hot water system
02 Ceatral warm air furnace with ducts to individual rooms, or
central heat pump (forced air)
03 Built-in electric units, permanently installed in wall, ceiling,
or baseboard
04 Floor, wall, or pipeless furnace
05 Circulating, radiant, or room heaters, WITH flue or vent,
burning gas, oil, or kerosene
06 Circulating radiant, or room heaters (not portable) WITHOUT
flue or vent, burning gas, oil, or kerosene
07 Portable room heaters of any kind
08 Fireplace(s) or stove(s) burning coal, wood, or coke
09 Some other type
SPECIFY:
10 No heating equipment
94 Don't know
8. Do you ever use air conditioning in your living quarters?
CIRCLE ONE NUMBER.
01 Yes, a central air-conditioning system
02 Yes, window unit(s)
03 No
9. Is there a garage attached to or within the structure in which you live?
CIRCLE ONE NUMBER.
01 Yes
02 No
-------
APPENDIX B
Page 5 of 12
10, Are your living quarters located Dear any of the following sites?
CIRCLE ONE NUMBER ON EACH LINE.
Yes
No
A.
Bus garage
, . 1 . .
. . 2
B.
Heavy vehicle maintenance area
or depot
. . 2
C.
Site of open burning
. . 1 . .
. . 2
D.
Electricity or steam plant . . ,
, . 1 . .
. . 2
E.
Manufacturing plant with heavy
smoke emission or power
generator or furnaces 1 .... 2
11. Do you travel to and from work, school, or any other place on a regular
basis at least three times a week?
CIRCLE ONE NUMBER.
01 Yes -ป CONTINUE
02 No - GO TO SECTION QUESTION 20.
The next set of questions is about the trips you make on a regular basis
during a nonsal week.
12. a. Please list in column A. in the table below each method of trans-
portation (car, bus, subway, walk, etc.) you use in commuting to
your destination. If you use more than one method, list them in the
order ia which you use them.
b. For each method you have listed in column A., enter in column B. the
total minutes you spend using this method to commute to your destina-
tion.
c. In column C., circle "Yes" or "No" to indicate if smokers are pre-
sent in each method of transportation you have listed.
A.
METHOD OF
TRANSPORTATION
B.
TOTAL MINUTES
SPENT
SMOKERS PRESENT?
Yes No
Yes No
Yes No
-------
APPENDIX B
Page 6 of 12
13. Do you return home by the same methods of transportation as those you
Listed in the table above?
01 Yes GO TO QUESTION 15.
02 No * CONTINUE.
14. a. Please list in column A. in the table below each method of trans-
portation (car, bus, subway, walk, etc.) you use in commuting from
your destination. If you use more than one method, list them in the
order in which you use them.
b. For each method you have listed in column A., enter in column B. the
total minutes you spend using this method to commute from your
destination.
c. In column C., circle "Yes" or "No" to indicate if smokers are pre-
sent in each method of transportation you have listed.
A.
METHOD OF
TRANSPORTATION
B.
TOTAL MINUTES
SPENT
Ci
SMOKERS PRESENT?
Yes No
Yes No
Yes No
15. At what time do you normally arrive at your destination?
CIRCLE ONE.
Time Arrive: AM PM
16. At what time do you normally depart for home?
CIRCLE ONE.
Time Depart: AM PM
17. About how many hours a week do you normally spend in an automobile either
as the driver and/or as a passenger? Include all time you spend com-
muting, shopping, on pleasure trips, etc.
Hours per week:
IF "NONE," GO TO QUESTION 20.
-------
APPENDIX B
Page 7 of 12
18. Please indicate the make, model, and year of Che automobile in which you
spend Che most time in a normal week.
MAKE:
MODEL:
YEAR:
19. Kow many of the hours you entered in Question 17 do you spend in the
automobile described in Question 18?
Hours per week:
20. Do you work either full-time or part-time outside your home?
CIRCLE ONE NUMBER.
01 Yes CONTINUE
02 No * GO TO QUESTION 32.
The next set of questions deals with the place at which you work.
21. Do you work at least part of the time in an enclosed area, other than in
a vehicle?
CIRCLE ONE NUMBER.
01 Yes CONTINUE.
02 No ฆ* GO TO QUESTION 28.
22. Which of the following best describes the enclosed area in which you work
the majority of the time?
01 Office or other small area, work alone
02 Office or other small area shared with other people
03 Large area (factory floor, laboratory, store, etc.), work
alone
04 Large area shared with other people
05 Circulate throughout building
06 Other (DESCRIBE)
23. What is the approximate si2e of the enclosed area in which you work the
majority of the time?
feet by feet
-------
APPENDIX B
Page 8 of 12
24. Is the enclosed area in which you work air-conditioned?
CIRCLE ONE NUMBER.
01 Yes, central air-conditioning system
02 Yes, window units(s)
03 No
25. Is a fan of any kind normally used in the enclosed area in which you
work?
CIRCLE ONE NUMBER.
01 Yes, window fan
02 Yes, other kind of fan
03 Yes, both window and other fan
04 No
26. In the enclosed area in which you work the majority of the time, are you
in the presence of smokers?
CIRCLE ONE NUMBER.
01 Yes, always (share work space with smoker(s))
02 Yes, most of the time
03 Yes, seldom
04 No
27. What is the main type of heating system at the place at which you work?
CIRCLE ONE NUMBER.
01 Steam of hot water system
02 Central warm air furnace with ducts to individual rooms, or
central heat pump (forced air)
03 Built-in electric units, permanently installed in wall, ceiling,
or baseboard
04 Floor, wall, or pipeless furnace
05 Circulating, radiant, or room heaters, WITH flue or vent,
burning gas, oil, or kerosene
06 Circulating radiant, or room heaters (not portable) WITHOUT
flue or vent, burning gas, oil, or kerosene
07 Portable room heaters of any kind
08 Fireplace(s) or stove(s) burning coal, wood, or coke
09 Some other type
+
SPECIFY:
10 No heating equipment
94 Don't know
-------
APPENDIX B
Page 9 of 12
28. Do you normally work near any of the following sites?
CIRCLE ONE NUMBER ON EACH LINE.
29.
Yes
No
A.
Bus garage
. . 1
. ... 2
B.
Heavy vehicle maintenance area
or depot
. . 1
. ... 2
C.
Site of open burning
. . 1
. ... 2
D.
Electricity or steam plant . .
. . 1
. ... 2
E.
Manufacturing plant with heavy
smoke emission or power
generator or furnaces ....
. . 1
. ... 2
a.
What kind of work do you do?
(For
example,
truck driver, etc.)
b. What kind of business or industry are you employed in? (For example,
TV and radio manufacturing, retail shoe store, U.S. government,
etc.)
30. How many hours do you normally work each week?
hours a week
31. How many of the hours that you work each week (Question 30) do you spend
outdoors?
hours a week
-------
APPENDIX B
Page 10 of 12
32. Please complete the table below concerning your leisure-time activities.
a. In column A., list up to five leisure-time activities at which you
spend the most time in an average week.
b. In column B. , describe the type of place where you participate in
each of the activities you listed in column A.
c. In column C., enter the approximate amount of time you spend at each
activity in an average week.
A.
ACTIVITY
B.
DESCRIPTION 01 PLACE
C.
TIME SPENT
hrs.
mins.
hrs.
mins.
hrs.
mins.
hrs.
mins.
hrs.
mins.
33. On shopping trips or any other leisure-time trips, do you normally park
in an indoor parking garage?
CIRCLE ONE NUMBER.
01 Yes - CONTINUE
02 No -< GO TO QUESTION 35.
34. In an average week, approximately how many total minutes do you spend in
an indoor garage when you are on leisure-time trips?
minutes
The last few questions are about you. Answers to these questions will be
used only in the statistical interpretation of the data.
35. What is your sex?
CIRCLE ONE NUMBER.
01 Male
02 Female
-------
APPENDIX B
Page 11 of 12
36. How old were you on your last birthday?
Age:
37. What is the highest level of education you completed?
CIRCLE ONLY ONE NUMBER.
01 Less than 8th grade
02 Some high school but did not graduate
03 High school graduate or equivalent
04 Some college but not a 4-year degree
05 4-year college degree
06 Post graduate work
07 Graduate degree
TK^NK YOU FOR YOUR COOPERATION. THE INTERVIEWER WILL COLLECT THIS QUESTION-
NAIRE WHEN HE OR SHE RETURNS FOR THE MONITOR AND ACTIVITY DIARY.
-------
APPENDIX B
Page 12 of 12
FI USE ONLY
Type of Housing Structure:
A. Constructed of:
1 Frame
2 Masonry (brick, cinder block, stucco)
3 Combination
B. Structure is:
1. One story house/duplex
2. Two- or more-story bouse/duplex
3. Split level
4. Garden-type apartment/condooinua
5. High rise apartment/condominum
6. Other (DESCRIBE)
FI No.
Date
Comments
-------
APPENDIX C
Page 1 of 6
EXAMPLE OF A TELEPHONE INTERVIEWER'S GUIDE
Source: "Study on Carbon Monoxide Exposure of Residents
of Washington, D.C., and Denver, Colorado,"
OMB Submission Packages Prepared for EPA by
Researcft Triangle Institute,
August 17, 1982
Household Screening Questionnaire
Hello. Is this (NUMBER)?
~ Yes -ป CONTINUE
~ No
t
What number have I reached?
THANK RESPONDENT AND TERMINATE.
My name is . I am calling from Research Triangle Institute in
North Carolina. We are calling randomly selected telephone numbers in the
Washington area in connection with a study we are conducting for the U.S.
Environmental Protection Agency. Is this a residential phone number?
~ Yes -+ CONTINUE
~ No
THANK RESPONDENT AND TERMINATE.
I need to speak to a resident of the household who is 16 years old/or older.
Are you a resident of the household?
~ Yes - CONTINUE
~ No
ASK TO SPEAK TO A RESIDENT WHO IS 16 OR OLDER. IF NO ONE IS AVAILABLE,
DETERMINE BEST TIME TO CALL BACK.
CALL BACK TIME A.M. OR P.M. DAY
Are you 16 years or older?
~ Yes -+ CONTINUE UNLESS RESPONDENT OBVIOUSLY CANNOT GIVE REQUIRED
INFORMATION; IF RESPONDENT IS INCAPABLE:
~ No
+ 11
ASK TO SPEAK TO ANOTHER RESIDENT WHO IS 16 OR OLDER. IF NO ONE IS AVAIL-
ABLE, DETERMINE BEST TIME TO CALL BACK.
CALL BACK TIME
A.M. OR P.M.
DAY
-------
APPENDIX C
Page 2 of 6
As I mentioned earlier, I am calling in connection with a study we are con-
ducting for the Environmental Protection Agency. The purpose of this study is
to determine to how much carbon monoxide people are exposed. In this part of
the study, we are getting information from approximately 5,000 households to
help us determine which people will be included in the final study. Your
participation is voluntary and the data that you provide will be kept con-
fidential. Any release of the data will be in summary form only. This part
of the study will take only a few minutes of your time.
1. a. First, what is the first name of the person who is the head of the
household?
ENTER BENEATH "01" IN COLUMN 1.
b. What are the first names of the other people who live in the house-
hold?
ENTER BENEATH HOUSEHOLD MEMBER NUMBERS IN COLUMN 1. DETERMINE
RESPONDENT AND INDICATE BY CIRCLING HOUSEHOLD MEMBER NUMBER IN
COLUMN 1.
c. I have listed (READ NAMES). Is there anyone else staying there now,
such as friends, relatives, or roomers?
IF YES, ADD TO ROSTER.
d. Do any of these people have a home somewhere else?
IF YES, DETERMINE WHICH PERSON(S) AND DELETE FROM ROSTER.
e. Do any of these people plan to move from this residence in the next
three months?
IF YES, DETERMINE WHICH PERSON(S) AND DELETE FROM ROSTER. IF
ENTIRE HOUSEHOLD PLANS TO MOVE, THANK RESPONDENT AND TERMINATE.
Now, I need a little information about each person.
ASK QUESTIONS 3-7 FOR HEAD OF HOUSEHOLD, THEN ASK QUESTIONS 2-7 FOR ALL OTHER
PERSONS ON ROSTER.
2. DO NOT ASK FOR HEAD OF HOUSEHOLD.
How is (NAME) related to (HEAD OF HOUSEHOLD)?
ENTER ONE OF THE FOLLOWING CODES IN COLUMN 2.
Spouse 02
Child 03
Other relative 04
Not related 05
ALSO ENTER "M" OR "F" IN COLUMN 3; ASK Q. 3 ONLY IF SEX NOT OBVIOUS
BY NAME OR DESCRIPTION.
3.
Is (NAME) male or female?
-------
APPENDIX C
Page 3 of 6
4. How old was (NAME) on (his/her) last birthday?
ENTER AGE IN COLUMN A.
5. Does anyone I have listed presently smoke, or use tobacco in any form?
IF "NO" - ENTER "N" IN EACH BOX IN COLUMN 5.
IF "YES" - PROBE AND ENTER "Y" OR "N" IN APPROPRIATE BOXES IN COLUMN
5.
6. Is (NAME) employed either full or part time?
IF "YES," PROBE FOR PRECISE OCCUPATION AND ENTER IN COLUMN 6.
IF "NO," DETERMINE WHAT PERSON DOES AND DESCRIBE IN COLUMN 6.
7. Does (NAME) travel to and from work, school, or any other place at least
three times a week?
IF "NO," ENTER ZEROES IN COLUMN 7.
IF "YES," ASK: How much time does (he/she) spend traveling one way?
ENTER IN COLUMN 7.
Now, a few final questions.
8. In your household, is there:
Yes No
a. a gas furnace 1 ... 2
b. a gas cooking stove 1 ... 2
c. a gas hot water heater 1 ... 2
d. a gas clothes dryer 1 ... 2
e. a gas or kerosene space heater . 1 ... 2
f. a fire place which is used ... 1 ... 2
g. a wood stove 1 ... 2
h. any other gas appliances .... 1 ... 2
SPECIFY:
-------
APPENDIX C
Page 4 of 6
9. Does your household have an attached garage, or share a raulti-family
garage?
~
~
Attached
Multi-family
~ Neither
10. Ia what city and county is your residence located?
City:
County:
Thank you very much for your help.
Based on the information that you and the rest of the 5,000 households in the
metropolitan Washington area provide, a random sample of approximately 500
individuals will be selected to participate in the main study. In addition,
500 households will be recontacted for some extra information. We hope that
you will be willing to help us if you are selected in either sub-sample. Have
a nice (day/evening). Goodbye.
PROVIDE THE FOLLOWING INFORMATION ONLY IF ASKED.
People who are selected for the main study will be asked to use a small port-
able monitor to record their exposure to carbon monoxide during a typical
24-hour period. They will be paid a small amount for completing this part of
the study.
-------
APPENDIX C
Page 5 of 6
1
2
3
A
5
6
7
House-
hold
Member
Relation
to
Head
Sex
Age
Smoker?
Occupation
Time Spent
Commuting
01
01
hr. min.
02
hr. ___ min.
03
hr. min.
04
hr. min.
05
hr. min.
06
hr. min.
07
hr. min.
08
hr. min.
09
hr. min.
10
hr. min.
-------
APPENDIX C
Page 6 of 6
APPOINTilEN'T:
AM
PM
Day of Week
Month
Date
Time
ADDRESS:
DIRECTIONS:
COMMENTS:
-------
APPENDIX D
Page 1 of 1
STANDARD PROCUREMENT LEADTIMES FOR CONTRACT AWARDS
(EPA Notice 84-3, 7/30/84)
The Procurement and Contracts Management Division announces new
standard procurement leadtimes for processing small purchase and
contract awards. Leadtime is defined as the elapsed time be-
tween receipt of an acceptable Purchase Request/Order (EPA Form
1900-8) in the appropriate Procurement Branch and the date of
the award. These leadtimes were established to reflect changes
ion award procedures as directed by the Federal Acquisition
Regulation and are effective immediately. Some actions may re-
quire extraordinary processing steps; others may require delayed
award dates to meet special needs of the program offices. In
such cases, special leadtimes will be negotiated with the ini-
tiators of the actions. We continue to encourage program offi-
ces to submit advance contract actions, as unfunded "planning
purpose procurements," as early as possible to ensure that the
award will be made by the desired contract date.
STANDARD PROCUREMENT LEADTIME
Category*: Leadtime in
Calendar Days
CONTRACTS:
A. New Competitive
Competitive - Research and
Development under $500K 185
Competitive - non R&D under 500K 170
Competitive - R&D $500K or more 240
Competitive - non R&D $500K or more .... 225
B. New Sole Source
New Sole Source - under $500K 120
New Sole Source - $500K or more 171
SMALL PURCHASES:
A. Competitive Order
Oral $1,000 to $10,000 5
Written $1,000 to $10,000 7*
Written $10,001 to $25,000 - non R&D 70
Written $10,001 to $25 ,000 - R&D 80
B. Sole Source Order
Order under $10,000 4
Written - more than $10,000 5*
Written - $10,000 to $25,000 60
Add 12 days if written quote obtained.
+ Federal Supply Schedule (FSS) orders will be included in these
categories based on the same criteria as other orders.
-------
CONTRACTS MANAGEMENT MANUAL
APPENDIX E
Page 1 of 4
PROCUREMENT REQUEST RATIONALE CHECKLIST
(to be submitted with EPA Forma 190G-8 and 1900-8A)
Item 1: The title of this procurement is
Iten 2t This procurement request package contains the following documentsซ
(Check all applicable boxes and attach documents as appropriate.)
Description
EPA Forms 1900-B
Procurement Abstract*
Statement or Scope of Work*
Conciue Technical Proposal Instructions*
Competitive Technical Evaluation Criteria*
Justification for Noncompetitive Procurement
(JNCP)*
Justification for Management Consulting
Services*
Justification of Need (Government-Furnished
Property (GFP) /Equipment)*
Quality Assurance (QA) Review Form
Recommended Sources List
Reports Description
Government-Furnished Property Description
* The PROJECT OFFICERS' HANDBOOK provides guidance for preparing these
documents. Also, see Item 11.
Item 3; This procurement /~7 requires / / does not require management
consulting services. (If management consulting services are required, attach
a justification as prescribed in EPA Acquisition Regulation 1537.205.)
Item 4: This procurement / / involves / / does not involve legal analysis. I
/ / have fl have not discussed this procurement with the Office of Legal and
Enforcement Counsel (QLEC) which / / concurs / / does not concur with
proceeding with this procurement.
Figure 2-1
(Page 1 of 4)
See Attachment # Check
N/A fj
fl
n
o
n
fl
n
/7
o
n
u
fl
-------
CONTRACTS MANAGEMENT MANUAL
APPENDIX E
Page 2 of 4
PROCUREMENT REQUEST RATIONALE CHECKLIST
Item 5: I / / anticipate or have knowledge of / / do not anticipate or have
any knowledge of organizational conflict of interests issues related to this
procurement. (If affirmative, describe conflict in an attachment.)
Item 6: Listed below are special EPA employee(s) who are or will be
participating in EPA's processing or managing of this procurement, together
with a list of their non-Government employers. Check here if none / /.
EPA Special Employees Non-Government Employer
Item 7; This procurement / / is / / is not based on an Unsolicitated Proposal.
Item 8; To the best of my knowledge the work results of this proposed
procurement / / are / / are not available from any other source. (If the
results are available from another source, describe in an attachment.) The
Project Officer / / has / / has not reviewed the Office of Pesticides and
Toxic Substance extramural activity report. The PO / / has / / has not
consulted the EPA Headquarters Library for relevant reports by previous
Contractors.
Item 9; The proposed Project Officer is .
He/she / / has / / has not been certified as an EPA Project Officer.
Item 1Q: I / / recommend / / do not recommend prospective sources for this
procurement. (If sources are recommended, list in an attachment.)
Item 11; This procurement anticipates / / a new contract award / / an
additional work modification to existing contract no. . It
also anticipates that it will be processed as a / / competitive procurement
/ / noncompetitive procurement. (If noncompetitive procurement is recom-
mended, attach appropriate justification as described in Part 1515 of the EPA
Acquisition Regulation.)
Item 12; This proposed procurement is appropriate for / / total small
business/labor surplus area (SB/LSA) set-aside; or / / partial SB/LSA
set-aside; / / partial SB set-aside; / / 8(a) set-aside; / / LSA set-aside;
or / / none of the above (check only one). (Consult the Office of Small and
Disadvantaged Business Utilization for advice.)
Item 13a; The estimated period of performance is months after the
effective date of the contract / / inclusive / / exclusive of submission of
any final report which may be required.
Figure 2-1
(Page 2 of 4)
-------
CONTRACTS MANAGEMENT MANUAL
APPENDIX
Page 3 of
PROCUREMENT REQUEST RATIONALE CHECKLIST
Item 13b: The schedule of deliverable items (excluding reports) is as
follows. Check here if no deliverable items are required / /.
Delivery
Item No. Description Quantity Date
Item 14; This procurement anticipates that the following options will be
needed. Check here if no options are anticipated / /.
Description of Option Term of Option
(Description may be indicated in a separate attachment)
Item 15ป The following reports are required (describe, in an attachment).
Check here if no reports are required / /. For each separate report
required, describe the following!
(a) Type of report (e.g., draft, final, interim, special, etc.)
(b) Descriptive title (e.g., monthly progress report)
(c) Minimum content requirements
(d) Number of copies required
(e) Distribution (with complete addresses of all recipients)
(f) Delivery schedule
(g) Number of days the Government will have to review, comment, approve
(disapprove) and return (as appropriate)
Where specific report formats, containing the information above, are used
repetitively, "standard" formats are established or may be established with
the servicing CO. Maximum use of such standard formats is encouraged.
Examples include monthly or other periodic progress reports, financial and
final reports.
Item 16: Peer review of Contractor-generated documents / / will be / / will
not be required.
Item 17; Government property, data, or services / / will bฎ furnished / /
will not be furnished under this procurement. (If furnished, describe in an
attachment including quantity and date available.)
Figure 2-1
(Page 3 of 4)
-------
CONTRACTS MANAGEMENT MANUAL
APPENDIX E
Page 4 of 4
PROCUREMENT REQUEST RATIONALE CHECKLIST
Item 18; Budget. (An attachment may be used.)
(a) The total estimated budget for the basic effort and all options is
$ .
(b) The estimated funding for the current fiscal year is $
(c) The estimated total cost of Other Direct Costs is $ .
(If possible, indicate estimate of significant subiterns such as
travel, computer time, consultants, equipment and material.)
(d) For level of effort actions and other actions where hours, rather
than an end product, are to be purchased, indicate for the basic and
all option periods the number of hours required, by category, with
definitions for each category.
Item 19: This procurement / / is / / is not subject to the requirements of
OMB Circular A-76. (If A-76 applies, required documentation oust be provided
with the PR.)
Item 20: This procurement / / requires / / does not require priority
processing (a brief priority justification may be detached).
(To be completed by procurement office:)
/ / Approved / / Disapproved
Date Chief, Contracting Office
Item 21: This procurement / / will / / will not involve the testing of human
subjects in accordance with EPA Order 1000.17.
Item 22: This procurement / / does / / does not include acquisition of
membership in an association. (If membership in an association is included,
attach a certification indicating that the primary purpose of membership Is
to obtain direct benefits for EPA necessary to the accomplishment of its
functions or activities.)
Item 23: This procurement / / is / / is not for leasing of motor vehicles.
(If affirmative, attach certification per FAR 8.1102.)
Item 24: This procurement / / is / / is not to be funded from more than one
appropriation. (If affirmative, see Chapter 9 of this manual.)
Figure 2-1
(Page 4 of 4)
-------
CONTRACTS MANAGEMENT MANUAL
APPENDIX F
Page 1 of 4
PROCUREMENT REQUS3T PPPROVJU^
A. Designated Office Approvals - Note that this matrix in Part A does not
restate any approvals from Part a. Offices designated in Part A should be
consulted prior to submission of .requests for approval .since authority may
have been redelegatad. The Agency objective is to limit Procurement
Request approvals strictly to those that are clearly required by proper
delegations and official policies.
Item Item Required Required
No. Description Local Approvals Headqaartars Approvals
1
1 All purchases, regardless (
of value, for equipment, 1
supplies or services.
(Includes all items
listed in the rest of ,
Part A.)
- Prorraic Office
App >ro v ป. 1 s ! V' OA)
- Fac-iiti.es
Services
Oi.vi.-Fi.on (FSSO)
j? ^< \i'7irr . i'
Fj.eli - iicp :ซSP0)
to comira cciis.pl i-
ance with property
regulations and to
screen for excc_js
property. (Dele-
gations Manual # 1-4)
None
2 Printing; duplication;
composition
Offices designated in
Delegation # 1-5 in
Delegations Manual.
None
3 - Real property or any
Interest therein
- Advertisements for
acquisition of real
property
None
FSSD
(Delegations Manual
# 1-4)
4 Commercial U-drive
credit cards
None
FSSD
(Delegations Manual
# 1-4)
5 Facsimile equipment
None
FSSD
(Delegations Manual
# 1-4)
Figure 2-2
(Page 1 of 4)
-------
CONTRACTS MANAGEMENT MANUAL
APPENDIX F
Page 2 of 4
PROCUREMENT REQUEST APPROVALS
Item Item Required Required
Nc. Description Local Approvals Headquarters Approvals
6 Communications eqaipiuent
utilizing the radio
frequency spectrum
None
FSSD
(Delegations Manual
# 1-4)
7 Watercraft over 25 feet
or $15,000 or aircraft
of any value
None
FSSD
(Delegations Manual
# 1-4)
..
8 Purchase or lease/hire
for 60 or more consecu-
tive days of passenger
vehicles and light duty
trucks
Hone
FSSD
(Delegations Manual
# 1-4)
9 Purchase, or rental
of copying machines
and printing equipment
None
FSSD
(Delegations Manual
# 1-5)
10 Repair and improve-
ment construction
FSSD or EFO
(See Delegations
Manual # 1-4)
FSSD
(For use of Building
and Facilities funds)
11 Equipment, services for
creation, organization,
maintenance and disposition
of Agency records and files
including inicrographic
services and systems
Limited offices as
designated in the
Delegations Manual
(see Delegation # 1-1)
See also Records
Management Manual.
None
12 &DP equipment, software
scaintsnance and services,
including those related
to computer-related
clcrographic systems,
word processing, time-
sharing, feasibility
studies and require-
ments analyses
None
Office of Information
Resources Management
(Delegations Manual
ป 1-10)
Figure 2-2
(Page 2 of 4)
-------
CONTRACTS MANAGEMENT MANUAL
APPENDIX F
Page 3 of 4
PROCUREMENT REQUEST APPROVALS
Item Item Required Required
No. Description Local Approvals Headquarters Approvals
13 Paid advertising for
Local personnel
None
recruitment of personnel
office (Delega-
in newspapers and trade
tions Manual
journals of national or
# 1-2)
inter-regional circulation
14 Collecting identical
None
Assistant Administrator
information or statis-
for Policy, Planning
tical data from ten
and Evaluation. (See
or more persons
Delegations Manual
~ 1-22)
15 Protective services and
Nor.t,
FSSD
equipment including guard
(See Delegations Manual
protection security alarms,
# 1-6)
safes, and monitoring and
detection devices
Figure 2-2
(Page 3 of 4)
-------
CONTRACTS MANAGEMENT MANUAL
APPENDIX F
Page 4 of 4
PROCDRZMBJT REQUEST APPROVALS
B. Management Approvals
The following approvals apply to all procurement requests (PRs).
(Incremental fundings are excluded.)
those listed in A above.
These approvals are in addition to
Item
Approval
1. Procurement Requests for
management consulting services:
(a) Small purchases
(b) Other than small purchases
Program official at least one
organizational level above
initiating office and, when
award is to be made during
the fourth fiscal quarter, to
program official at least two
organizational levels above
initiating office.
Program official not below
level of Associate, Assistant
or Regional Administrator,
Inspector General, or General
Counsel.
2. Requirements not listed above
which exceed the small purchase
threshold.
Program Official designated by
the Associate, Assistant or
Regional Administrator,
Inspector General, or General
Counsel.
Figure 2-2
(Page 4 of 4)
^U.S, GOVERNMENT PRINTING OFFICEi 1 9 a 5 - 461 *SiI/2ซ0 J J
------- |