EPA-810/B-92-015
United States Office of Water (WH-550) EPA 810-B-82-015
Environmental Protection Office of Pesticides and February 1992
Agency Toadc Substances (H-7501C)
QUALITY ASSURANCE PROJECT PLAN
FOR THE
NATIONAL PESTICIDE SURVEY OF DRINKING WATER WELLS
SURVEY STATISTICS, DATA COLLECTION AND PROCESSING
Prepared by:
ICF Incorporated
9300 Lee Highway
Fairfax Virginia 22031
and
Westat Corporation
1650 Research Boulevard
Rockville, Maryland 20850
Prepared for:
U.S. Environmental Protection Agency
Technical Support Division
Office of Drinking Water
26 W. Martin Luther King Drive
Cincinnati, Ohio 45268
-------
Section No. 1
Revision No. 3
Date: March 14, 1990
Page 2 of 2
APPROVAL PAGE
Harold Lester Gary McKown
ICF Project Director ICF Quality Assurance Officer
David Marker David Morganstein
Westat Project Director Westat Quality Assurance Officer
Elizabeth Leovey Maria Gomez-Taylor
EPA Quality Assurance Officer EPA Quality Assurance Officer
Office of Pesticide Programs Office of Drinking Water
Jean Briskin
NPS Project Director
-------
Section No 2
Revision No 0
Date: March 14, 1990
Page 1 of 1
QUALITY ASSURANCE PROJECT PLAN
FOR THE
SURVEY STATISTICS, DATA COLLECTION AND PROCESSING
2. TABLE OF CONTENTS
Section
1. TITLE AND APPROVAL PAGE
TABLE OF CONTENTS
2.
3.
4.
5.
6.
PROJECT DESCRIPTION
3.1 Data Collection Design and Timing
3.2 Improving Quality Versus Measuring Quality
PROJECT ORGANIZATION AND RESPONSIBILITIES
COMMUNITY WATER SYSTEM SURVEY TASKS
AND QUALITY ASSURANCE PROCEDURES
5.1 Community Water System Screening Survey
5.2 Community Water System Field Survey
DOMESTIC WELL SURVEY TASKS AND
QUALITY ASSURANCE PROCEDURES
6.1 Second-Stage County Agent Interview
6.2 DWS Screening Survey
6.3 Domestic Water System (DWS) Field Survey
Appendices
A. CWS SCREENING SURVEY TRAINING MANUAL
B. BLANK COPIES OF THE CWS QUESTIONNAIRE,
CWS TEAM LEADER INTRODUCTION AND WELL
OBSERVATION RECORD, AND LOCAL AREA
QUESTIONNAIRE
C. CALCULATION OF CLUSTER SPECIFICATIONS
D. STANDARD OPERATING PROCEDURES
E. DOMESTIC WELL SCREENING SURVEY TRAINING
MANUAL
F. DOMESTIC WELL SURVEY INTERVIEWER
TRAINING MANUAL
Pages
2
1
2
22
23
1
1
3
Revisions
0
0
0
0
0
0
Date
3/14/90
3/14/90
3/14/90
3/14/90
3/14/90
3/14/90
3/14/90
3/14/90
3/14/90
3/14/90
3/14/90
3/14/90
-------
Section No. 3
Revision No. 0
Date: March 14, 1990
Page 1 of 2
3. PROJECT DESCRIPTION
The National Pesticide Survey (NPS) is designed to meet two major objectives: (1) to determine
the frequency and concentration of pesticide contamination in the drinking water wells of the nation,
and (2) to better understand how pesticide contamination is associated with patterns of pesticide
usage and the vulnerability of groundwater to pollution. A statistical design ensures that the wells
sampled represent 13 million private domestic wells and all 51,000 community well systems
throughout the U.S. Approximately 1,350 domestic and community wells will be surveyed between
1988 and 1990.
This QAPjP outlines the Survey statistics, questionnaire data collection and processing
procedures for community and domestic drinking water wells. For both the Community Water System
(CWS) and Domestic Well Surveys these procedures involve developing and implementing
methodologies to select approximately 1,350 wells for sampling and questionnaire administration. Due
to the inherent differences in domestic and community water wells, the procedures for data collection
and processing are slightly different. The data collection and processing procedures for the CWS
drinking water well Survey are addressed in Chapter 5 and those for the Domestic Well Survey are
addressed in Chapter 6.
Section 5.1 outlines the tasks of sample selection; questionnaire preparation; questionnaire
administration; data preparation; and selection of field sample for the screening portion of the CWS
Survey. Section 5.2 outlines the tasks of questionnaire preparation; interviewer training; materials
preparation and mailout; monitoring of field activities and questionnaire receipt; initial editing and initial
data retrieval; data preparation and data entry; and data reduction, analysis, and reporting for the field
portion of the CWS Survey.
Section 6.1 briefly describes the Second-stage County Agent Questionnaire interview. Section 6.2
outlines the tasks of questionnaire preparation; preparation of county-specific questions; sample
selection; questionnaire administration; and selection of field sample for the screening Survey portion
of the Domestic Well Survey (DWS). Section 6.3 outlines the tasks of questionnaire preparation;
interviewer training; questionnaire receipt; questionnaire administration; initial editing and initial data
retrieval; data preparation and data entry; data reduction, analysis, and reporting; and systems
security and data backup and archival for the field portion of the Domestic Well Survey.
Included at the end of this QAPJP are appendices that contain Standard Operating Procedures,
copies of relevant questionnaires, and training manuals.
3.1 Data Collection Design and Timing
The CWS data collection effort will consist of two main parts: a screener Survey and the main
Survey. The screener will consist of short telephone interviews in early 1988 with approximately 7,000
-------
Section No. 3
Revision No. 0
Date: March 14, 1990
Page 2 of 2
CWSs identified from the Federal Reporting Data System (FRDS). From the results of this screener a
stratified sample of approximately 600 eligible CWSs will be drawn for participation in the main survey.
The main data collection effort will be conducted from late summer 1988 until the end of calendar year
1989. Westat will draw both samples, conduct the screener Survey, and assist ICF in training
interviewers and samplers for the main Survey.
The domestic well data collection effort will also consist of a screener and a main Survey. Westat
will conduct a random digit dialing (ROD) screener in 84 counties to identify eligible households with
wells. Westat interviewers will conduct the interviews in the main Survey and assist ICF in sampling
the domestic water wells. Unlike, the CWS, the DWS screener and main Survey are conducted in
parallel. The main Survey will be in the field from the spring of 1988 until early 1990, with the screener
Survey (except for the six pilot counties) generally taking place one to two months in advance of the
field work for any particular county.
3.2 Improving Quality Versus Measuring Quality
For survey errors, there are two philosophies to consider. In Groves' book Survey Errors and
Survey Costs (John Wiley & Sons, 1989) he compares,"... efforts to reduce the various errors afflicting
surveys and efforts to measure the errors. On the side of reduction are scores of efforts to train
interviewers...examples of question wording that improves the quality of the information collected, and
efforts to improve response rates...." "In sharp contrast to this research [are studies]...designed to
estimate their magnitudes."
In implementing the NFS, ICF and Westat will concentrate efforts on improving quality rather than
measuring its absolute level. (For some activities it also will be possible to estimate errors, e.g.
response rates, but our priority will be on reducing errors.) Thus our QA activities will deal with
minimizing errors at every stage of the process, whether it is selecting a sample, training interviewers,
or processing the data. This Quality Assurance Project Plan describes the efforts undertaken to
minimize errors and increase quality as part of Survey statistics, data collection, and processing
activities.
-------
Section No. 4
Revision No. 0
Date: March 14, 1990
Page 1 of 7
4. PROJECT ORGANIZATION AND RESPONSIBILITIES
Westat's National Pesticide Survey (NPS) Project Management organization is diagrammed in
Exhibit 4-1. Mr. Harold Lester is ICF's Responsible Officer-in-Charge for the NPS. Mr. David Marker is
the Westat Project Director, responsible for all of Westat's activities for NPS. Mr. Marker manages
Westat's staff activities. As directed by ICF, the prime contractor, Westat responsibilities include four
key tasks:
(1) Developing and implementing the statistical selection of domestic and community wells
for the NPS;
(2) Providing and supervising trained, professional interview staff for the NPS;
(3) Developing and implementing data processing procedures for all questionnaires; and
(4) Assisting EPA and ICF in analyzing Survey data.
Mr. David Morganstein is the Westat Quality Assurance Officer, responsible for reviewing and
approving the Quality Assurance Plan. Ms. Leslie Athey and Mr. David Marker are Survey Operations
Managers, responsible for Survey operations. They supervise all questionnaire development and
training, interviewing, and questionnaire data processing activities. Mr. Morganstein will conduct
internal audits of Westat's data processing activities. All Westat personnel can be contacted at 1650
Research Boulevard, Rockville, Maryland, 20850; or by calling (301) 251-1500.
Ms. Lora Johnson, NPS QA Officer, will conduct periodic audits of Westat's activities as part of
EPA's quality assurance reviews.
-------
Section No 4
Revision No 0
Date: March 14, 1990
Page 2 of 7
Exhibit 4-1
Westat Project Management Organization
Steve Diet*
Project Director
David Marker
Stattatlcal Staff
David Marker
Leatte Atney
9A
David
Earl Brymnt
Huaeytn Gokael
Joe Wakaberg
David Wright
CoounoDitv Water
Svateflia
Receipt Control.
Data Entry
LlaaPohl
Klraten Schroeder
LeaUe Athey
CWS Traiaiac aad
Implementation
Leatte Atbey
Rotraut BockatahJer
Tom MUke
Joyce Powell
Klreten Schroeder
Steve SUJer
Pal Stoufler
TRC Sla/I*
LynnCUrk
Oavld Hurwltz
Joyce Powell
Pat Proulx
UaaPuhl
DougDoucui
Janice Machado
AnnMaMe
Tom Mllke
Un MUter
Judy Rawnucwn
Gene Robtnaon
Mike Scott
Suaan Tenney
Coder*
Data Entry Staff
TRC Staff*
SpecifleatkM
LeaMe Atber
Vicky Albright
Norm BrauOgam
5ewny Haiuuian
Grethel Hoflmalater
JubeUm
Ann Maaae
BUI Rtehey
Sandy Cabrera
Suaan Hanaon
Dennla Optermana
Oaud Santoaa
Klraten Schroeder
TRC Staff*
• Telephone Research Center Staff
Laura Brow
OrtDtetrtc
MarleneEd
Alice Faby
Virginia FtM
Erna CUlec
Sandy Klrtr
Diane Und<
Barbara Sli
-------
Section No. 4
Revision No. 0
Date: March 14, 1990
Page 3 of 7
Exhibit 4-1 (continued)
Westat Project Management Organization
CWS Screener
Leslie Athey
Screener Sample
Selection
David Marker
Telephone
Screening
Rotraut Bookstahler
Screener Coding
mnit nmt» g»if|y
LeatteAtbej
Earl Bryant
JocWatoberg
Klrsten Schroeder
TRC Staff
Tom MUke
Joyce Puwdl
Steve sola-
Pat Stoufler
-------
Section No. 4
Revision No 0
Date March 14, 1990
Page 4 of 7
Exhibit 4-1 (continued)
Westat Project Management Organization
CWS Training and
Implementation
Leslie Athey
CWS State Training
Leslie Athey^
CWS
Implementation
UsaPuhl
Lynn Clark
Pat Proulx
Usa Puhl
CWS Tracking
System
TomMilke
David Hurwttz
Joyce Powell
CWS Sampling
David Marker
Lisa Puhl
Mtke Scott
Joe Waksberg
-------
Section No 4
Revision No 0
Date- March 14, 1990
Page 5 of 7
Exhibit 4-1 (continued)
Westat Project Management Organization
Receipt Control,
Coding and Data
Entry
UsaPuhl
Receipt Control
Coding and COED
CWS Receipt
Control
UsaPuhl
DWS Receipt
Control
Swa
Amtc
Data Entry
Judy Rasmuaaen
DataRetxieti
Janice M««»imi
Data Entry Staff
CWS
and COED
UsaPuhl
David Hurwiu
Tom Milke
Gene Robinson
Mike Scott
Susan Tennev
Rotraut Bodcst&f
UsaPuhl
TRC Staff
DWS
and COED
UsaPuhl
Janice Machado
Tom Mllke
Mike Scott
Doug Duncan
Janice Machado
Len Miller
Diane Sickles
Susan Tenney
Coders
Doug Duncan
Janice Machado
Ann Masse
Len Miller
Diane Sickles
Susan Tenney
Coders
-------
Section No. 4
Revision No. 0
Date: March 14, 1990
Page 6 of 7
Exhibit 4-1 (continued)
Westat Project Management Organization
DWS Mapping and
Clutter Spec*
Leslie Athey
Cluster Spec*
Leslie Atkey
Procedure
Leslie Athey
Map Boundaries
Vicky Albright
County Questions
Derelopment ft
Testing
Ann Masse
CAT! Programming
Norm Bisuftgsm
Vicky Albright
Grethel Hoffmaister
Ann Masse
David Marker
Beverly Hardman
Grethel Hoffmaister
Ann Masse
Bill Rlchey
Vicky Albright
Beverly Hardman
Bill Rlchey
Julie Llm
-------
Section No. 4
Revision No. 0
Date- March 14, 1990
Page 7 of 7
Exhibit 4-1 (continued)
Westat Project Management Organization
DWS RDD Screener
Ann Masse
Cluster
Identification
Sandy Cabrera
Dally Releases
Ann Masse
Conducting
Interviews
Susan Hanson
Dennis Opfermann
Daud Santosa
TRC Staff
Klrsten Schroeder
TRC Staff
-------
Section No. 5
Revision No. 0
Dale: March 14, 1990
Page 1 of 22
5. COMMUNITY WATER SYSTEM SURVEY TASKS AND QUALITY ASSURANCE PROCEDURES
5.1 Community Water System Screening Survey
The purpose of the Community Water System (CWS) Screening Survey is to determine the
eligibility of CWSs for the field Survey. Eligibility for the field Survey requires that a CWS:
Serve piped drinking water;
Serve 25 or more permanent residents and/or have 15 or more connections; and
Have at least one working well used for drinking.
The CWS Screening Survey is performed in five discrete steps as outlined below: Sample
Selection, Questionnaire Preparation, Questionnaire Administration, Data Preparation, and Selection of
Field Sample.
5.1.1 Sample Selection
The objective of sample selection for the screening Survey is to identify a national sample of
CWSs that meet the selection criteria for eligibility outlined above with a small probability of error. EPA
provided Westat with the CWS Screening Survey sampling plan. Westat will be responsible for
reviewing and implementing the plan.
5.1.1.1 Summary of Procedures
The sample frame for the CWS Screening Survey is the Federal Reporting Data System (FRDS),
maintained by Planning Research Corporation. First, EPA identified in FRDS over 50,000 CWSs using
groundwater as a source of drinking water. Using this listing, EPA stratified the CWSs by pesticide
usage (i.e., high, moderate, low and uncommon) and ground-water vulnerability (i.e., high, moderate,
and low) of the county in which each CWS is located. A sample of 7,000 CWSs from the FRDS will be
randomly selected for the CWS Screening Survey with oversampling of those CWSs that are in the
high ground-water vulnerability stratum. A subsample of approximately 600 CWSs will then be
randomly selected from the eligible CWSs identified out of the 7,000 for participation in the drinking
water sampling Survey. (The CWS sample design is further described in the National Survey of
Pesticide Residues in Community Water Systems and Rural Domestic Wells: Sample Allocation Report,
Research Triangle Institute, April 6, 1988, Section 4.2.)
Exhibit 5-1 summarizes the distribution of samples across all strata and reflects the oversampling
of CWSs in high ground-water vulnerability strata. The systematic random sample within each stratum
for the CWS Screening Survey is selected using the respective sampling rates and starting points.
-------
Section No. 5
Revision No. 0
Date: March 14, 1990
Page 2 of 22
Exhibit 5-1
Distribution of Community Water System Sample Size
Stratun
1
2
3
4
5
6
7
8
9
10
11
12
Total
Pesticide
Usage
High
High
High
Medium
Medium
Medium
Low
Low
Low
Uncommon
Uncommon
Uncommon
Groundwater
Vulnerability
High
Medium
Low
High
Medium
Low
High
Medium
Low
High
Medium
Low
Sample
Size
292.9
276.7
148.4
555.9
355.3
265.2
867.3
772.0
646.6
776.2
1,068.9
1.060.9
7,086.3
Universe
Size
1,310
2,669
1,432
2,183
3,352
2,559
3,578
7,199
5,545
3,472
9,511
7.961
50,771
Sample
Fraction
0.2236
0.1037
0.1032
0.2546
0.1060
0.1036
0.2424
0.1072
0.1166
0.2236
0.1124
0.1333
0.1396
Sampling
Rates
4.47
9.65
9.65
3.93
9.43
9.65
4.13
9.33
8.58
4.47
8.90
7.50
Starting
Points
1.01
6.90
6.24
3.42
7.85
4.14
1.51
8.27
0.84
1.91
5.58
3.54
Using the stratified, systematic random sample method, the universe of CWSs for which
groundwater is the source of drinking water is first subdivided into strata. A systematic random
sample is then drawn within each stratum. In stratum 1, for instance, the sequence is 1.01, 1.01 +
4.47, 1.01 + 2x4.47, 1.01 +3x4.47, etc. Each value is rounded up to the nearest whole number to
obtain the sequence number of the respondent within the stratum. This sampling is continued until
the list is exhausted.
5.1.1.2 Quality Assurance and Quality Control Procedures
The quality assurance and quality control procedures for sample selection entail reviewing the
sampling plan and its implementation and automation and reviewing the automated sample selection
results as follows:
Reviewing the Sampling Plan - The Westat Project Director, an experienced sampling
statistician, reviews EPA's design of sample selection procedures. It is further reviewed
by the previous head of the Statistics Study Area, a Westat Senior Vice President.
• Implementing the Sampling Plan - The Westat Project Director designs step-by-step
procedures to be followed in the implementation of the sampling procedures.
Automating the Sampling Plan - The Westat Senior Systems Analyst, an experienced
programmer, automates the sample selection procedures.
• Reviewing the Automated Sample Selection - Sample results of the automated
programs are checked exhaustively by the Westat Project Director to confirm the
accuracy of the programming.
-------
Section No. 5
Revision No. 0
Date: March 14, 1990
Page 3 of 22
5.1.2 Questionnaire Preparation
The objective of questionnaire preparation is to ensure that the CWS Screening Survey
Questionnaire is worded and formatted such that it collects the information specified by EPA to satisfy
the objectives of the CWS Screening phase of the Survey (Section 5.1).
5.1.2.1 Summary of Procedures
The original CWS Screening Survey Questionnaire design was prepared by EPA and supplied to
Westat. This questionnaire had been pretested during the pilot phase of the National Pesticide
Survey. Westat's role in this task will be to review and reformat the questionnaire. Specifically, this
screening questionnaire is designed to collect the information required to make a determination of
CWS eligibility, the number of wells, and the appropriate CWS contact.
5.1.2.2 Quality Assurance and Quality Control Procedures
The CWS Screening Survey Questionnaire will be prepared by Westat staff and reviewed by in-
house experts for appropriate wording and formatting. The questionnaire will be reviewed by EPA and
ICF prior to implementation.
5.1.3 Questionnaire Administration
The objective of questionnaire administration is to execute the Survey in a manner which follows
established Survey protocol. The CWS Screening Survey will be administered using a paper
questionnaire. The respondent for the CWS Screening Survey will be the CWS contact identified
during the sample selection task 5.1.1.
5.1.3.1 Summary of Procedures
The procedure for administration of the CWS Screening Survey Questionnaire consists of phone
interviewing and respondent tracking. In order to conduct the actual screening interview, the Westat
interviewer will use the Respondent Information Sheet (RIS) (containing the address, phone number
and other CWS owner information obtained from FRDS) and the CWS Screening Survey
Questionnaire. The interviewer will call the respondent listed for the CWS and administer the CWS
Screening Survey Questionnaire. The procedure for conducting the CWS screening interview is
described in the interviewer training manual, which is included as Appendix A. The results of each
interview will be entered into a file from which weekly progress reports will be generated. Since many
of the names and phone numbers may be incorrect, incomplete or not in service, considerable effort
in tracking the respondent may be required. For specific cases, tracking may include contacting
Directory Assistance, or local government agencies or businesses.
5.1.3.2 Quality Assurance and Quality Control Procedures
The quality assurance and quality control goal for administering the CWS Screening Survey
Questionnaire is to design and implement a Survey control system which monitors the consistency of
-------
Section No. 5
Revision No. 0
Date- March 14, 1990
Page 4 of 22
questionnaire administration and ensures that the Survey objectives are met. Westat will institute the
following practices to ensure the quality of the CWS Screening Survey Questionnaire and its
administration:
Training interviewers - Interviewers will receive at least six hours of Survey-specific
training given by the Deputy Project Director and telephone management staff. The
training will include question-by-question discussion of the questionnaires and role
playing. Each interviewer will be provided with a manual to assist with training and to
serve as a reference during interviews.
• Conducting interviewer evaluations during training - Interviewers will be observed by
the Deputy Project Director and telephone center management staff during the training
role plays. Interviewers showing difficulty in understanding or administering the
questionnaires will be given additional training until they demonstrate an acceptable
skill level. In extreme cases, interviewers may be dismissed from the study.
Monitoring interviewers by supervisory and project staff - Supervisors silently monitor at
least five percent of the randomly selected telephone interviews. Interviewers will not
know when their interviews are being monitored. Interviewers showing difficulty in
understanding or administering the questionnaire will be given additional training until
they demonstrate an acceptable skill level. In extreme cases, interviewers may be
dismissed from the study. A copy of Westat's interviewing monitoring form is in
included as Exhibit 5-2.
Providing continuous availability of experienced supervisors - Telephone supervisors
will be on the interviewing floor near the interviewers during all interviewing work.
• Reviewing non-locatable cases - Supervisors will review all non-locatable cases
prior to listing a final code of non-locatable to assure that all reasonable efforts
have been made to contact the CWS listed on the FRDS file.
Maintaining problem logs (by telephone supervisors) - Telephone supervisors will
maintain logs of all problems encountered and conduct frequent meetings with
interviewers and project staff to review problems. Corrective action strategies will be
developed to minimize the reoccurrence of these problems.
Producing weekly production reports - Weekly reports will be produced to monitor the
status of the CWS screening operation. Exhibit 5-3 is a copy of Westat's Telephone
Research Center Weekly Production Report. The report contains both weekly and
cumulative information on interviewing hours, time per completed case, response rates,
and a detailed breakdown of cases by result status.
• Reviewing 100 percent of completed questionnaires - All completed interviews will be
reviewed by the original interviewers and by clerical staff. The interview supervisor will
review problem interviews.
Conducting formal debriefing of interviewers - A formal debriefing session will be held
with interviewers to identify problems that were encountered while conducting
interviews.
-------
Section No. 5
Revision No. 0
Date: March 14. 1990
Page 5 of 22
Exhibit 5-2
Monitoring Form
cart
C4TI
ri«
T!«
Mm
tt
a rou ••nitor
CdiM4*c* t
3tn«r w««
int«rvt«M I
•f
»rm-
rtai
MOVIOI a.«
ItMIM MB QOOM. «ILU
Ml M,1
atui «• MOIIM
t* «mir« MMMT
« Usura tme
«wt M MI«
-------
Section No. 5
Revision No 0
Date: March 14, 1990
Page 6 of 22
Exhibit 5-2 (continued)
Monitoring Form
IfCOBIM OATA
. UM. _*. «~M.t..r .
4 CATPVBYS 6flBtA9 vrrvpv
< UMB c«icr«t «•*» if Mil tv
1M0I.1M ItMULS AM 4UISTIOM
ctvwtv «• arivttr »ttim*
•• «tu*r
. ».«i —.,«,- —
-T;™!!™
!^rjii,mir "ri**4
• ItMMi OOC^n^ MTMnBUT
I/A
s-
ATTU*
TtOi
•
(tr MOOLOI WTO. MOVtOC 9.4 «M Og»
-------
Section No. 5
Revision No. 0
Date: March 14. 1990
Page 7 of 22
Exhibit 5-3
Telephone Research Center Weekly Production Report
:FA - 'isr;^iL -ESTICIDE JTUDY DRINKING MTER «us)
'ElE-HONE Jf'EAfCH CENTER KEKLY PRODUCTIOH REPORT
:ATE :F
TO:
CETERKIW B» C»PCM+SI*S2+S;/T5TAL JA«PLE ««cJuain$ lOIKaCATABLES
-------
Section No. 5
Revision No. 0
Date: March 14, 1990
Page 8 of 22
5.1.4 Data Preparation
The objectives of data preparation are to produce a computerized questionnaire data file in a
manner that minimizes errors and to provide for the physical security of the data. Data preparation
consists of coding, keying, editing, data retrieval, and providing data security.
5.1.4.1 Summary of Procedures
The CWS Screening Survey questionnaire is pre-coded wherever possible to facilitate key entry.
All completed questionnaires will be reviewed by a clerical staff member and then by a coder. Key
entry will be performed with data entry scree/is using software that checks for out-of-range codes
before storing the information in the database. Completed work will be secured and backed-up.
Access to the data and computers is controlled and all personnel with access to this information will
sign a confidentiality pledge. Back-up files are then stored and secured.
5.1.4.2 Quality Assurance and Quality Control Procedures
The quality assurance and quality control goal for data preparation is to ensure that the data are
entered into a computer database and secured in a manner that will not compromise the integrity of
the data. Coding, data editing, and data entry supervisors will be available when these activities are
occurring to solve problem situations.
Specifically, the following procedures have been developed to ensure the accuracy and
consistency of the data coding activities:
Validating selected cases by coding supervisors - Coding supervisors will completely
re-code a selected sample of cases. Validation coding will be conducted throughout
the Survey. More frequent validation will be performed more in the early phases of the
coding process. Coders demonstrating difficulty in understanding or completing tasks
will be retrained. In extreme cases, coders may be dismissed from the Survey.
• Verifying numerical data on selected cases - The numerical data will be key-verified on
at least five percent of the cases. Discrepancies between the originally keyed data and
subsequently keyed data will be resolved by the data entry operator performing the
verification keying. Data entry operators demonstrating difficulty in understanding or
completing tasks will be retrained. In extreme cases, data entry operators may be
dismissed from the Survey.
Maintaining problem logs - Coding supervisors will maintain logs of all problems
encountered and conduct frequent meetings with coders and project staff to review
problems. Corrective action strategies will be developed to minimize the reoccurrence
of these problems.
• Conducting frequent meetings - Frequent meetings will be conducted between coders,
data entry operators, supervisors, and project staff to ensure that any problem
encountered is resolved quickly and that solutions are communicated to the staff
readily. These meetings will be arranged on an "as needed" basis, not at regularly
scheduled intervals.
-------
Section No. 5
Revision No. 0
Date: March 14, 1990
Page 9 of 22
In order to ensure the security of collected data, Westat will institute the following procedures:
• Limited access - The RCRA Confidential Business Information (CBI) PC processing
facility will be used. The PC will be run in a lockable office with limited access by
personnel.
Physical security - All files will be backed-up and kept in a separate storage area.
Create permanent archive - When data are published, the raw data, methodology and
procedures will be archived at ICF in order to ensure that the published results are
reproducible.
5.1.5 Selection of Field Sample
The objective of the selection of the field sample is to select a sample of CWSs that meet the
eligibility criteria with a small probability of error and represent CWS wells throughout the US. Westat
is responsible for developing, reviewing, and implementing the CWS well selection sampling plan and
delivering the eligible CWS sample to ICF.
5.1.5.1 Summary of Procedures
The CWS Screening Survey will produce a frame from which to select the CWS field sample.
From the 7,000 CWSs to be screened a stratified sample of roughly 600 eligible systems will be
selected. Within each stratum, the CWSs will be sorted by State and number of wells.
Exhibit 5-4 summarizes the distribution of sampling rates across strata. The starting point is a
random number between 0 (rounded up to the nearest .01) and a cutoff which is defined as the total
number of eligible wells divided by the sample size in that stratum.
Sample characteristics will provide the basis for developing weights during the analysis of the
Survey results and provide population characteristics that will assist the analyst in imputation for
missing data. This also will provide information on the quality of the FRDS database, which is the
sampling frame for the Survey.
5.1.5.2 Quality Assurance and Quality Control Procedures
Westat will employ quality assurance and quality control procedures for the selection of field
sample operations similar to those for CWS screening (Section 5.1.1.) as follows:
• Automating sampling procedures - A Senior Systems Analyst will prepare automated
procedures.
Programming accuracy checks - The automated program selections will be confirmed
by the sampling statistician to ensure that each selected CWS was selected correctly.
Monitoring completion rates - The Westat Project Director and ICF staff will monitor
refusals and ineligible systems through the sampling period to ensure that sufficient
completion rates are maintained. If completion rates are significantly below
expectations it may be necessary to select supplemental samples from the eligible
screened systems.
-------
Section No. 5
Revision No. 0
Date: March 14, 1990
Page 10 of 22
EXHIBIT 5-4
DISTRIBUTION OF SAMPLES ACROSS STRATA
Stratum
1
2
3
4
5
6
7
8
9
10
11
12
Pesticide
Usage
High
High
High
Medium
Medium
Medium
Low
Low
Low
Uncommon
Uncommon
Uncommon
Groundwater
Vulnerability
High
Medium
Low
High
Medium
Low
High
Medium
Low
High
Medium
Low
Sample
Size
27
26
14
45
32
25
73
68
53
71
90
75
599
Take
Every
n/27
n^/26
rig/14
n4/45
n5/32
n6/25
n7/73
n8/68
ng/53
n10/71
n^/90
n12/75
Where n is the number of eligible CWSs in the stratum.
5.2 Community Water System Field Survey
The purpose of the CWS Field Survey is to collect and analyze information pertaining to the
CWSs from which water samples are drawn. This information is collected through the administration
of three questionnaires: the CWS Questionnaire, the CWS Team Leader Introduction and Well
Observation Record, and the Local Area CWS Questionnaire. Examples of these questionnaires
appear in Appendix B.
5.2.1 Questionnaire Preparation
The objective of questionnaire preparation is to ensure that the instruments used in the CWS
Field Survey are worded and formatted such that they collect the information specified by EPA to
satisfy the objectives of the CWS Field Survey. Specifically, the information that is gathered includes,
but is not limited to, depth of the well, well construction characteristics, water treatment, depth to
groundwater, general aquifer characteristics, soil texture in the immediate vicinity of the well, and
pesticide usage in the vicinity of the well. Westat will modify the Survey instruments developed from
the pilot phase of the NPS and incorporate EPA's and ICF's comments. Westat will review and
reformat the questionnaires as necessary to minimize administration errors.
-------
Section No. 5
Revision No. 0
Date: March 14, 1990
Page 11 of 22
5.2.1.1 Summary of Procedures
The CWS Questionnaire and CWS Team Leader Introduction and Well Observation Record will be
completed in the field at the time of well water sampling. The respondent for the CWS Team Leader
Introduction and the CWS Questionnaire is the CWS representative. The CWS Team Leader
Introduction will be designed to introduce the CWS representative to the Survey by presenting an
overview of the Survey objectives. Its most important function will be to identify through a random
selection process the specific well to be sampled in cases where the CWS is a multiple well system.
The CWS Questionnaire will be designed to collect information regarding water usage, water
treatment, and well construction.
The CWS Well Observation Record will be completed by EPA Region, State, or ICF field team
members during well water sample collection activities. This instrument will be used by the field team
to formally record information regarding soil characteristics and other significant features in the vicinity
of the well.
The Local Area CWS Questionnaire will be administered to a person familiar with local conditions
and usage of land and pesticides surrounding the CWS being sampled. This person will usually be
the County Agricultural Extension Agent. This interview should be conducted within one week of the
well water sampling activities.
5.2.1.2 Quality Assurance and Quality Control Procedures
The objectives of the quality assurance and quality control procedures for the CWS field Survey
questionnaire preparation are to ensure that these instruments are unambiguous and easy to follow
and that they accurately and completely collect the information requested by EPA to satisfy the
objectives of the CWS Field Survey outlined above. The original CWS questionnaires were prepared
and pre-tested by EPA and supplied to Westat. The questionnaires will be reworded and reformatted
by Westat, and reviewed by in-house experts. The questionnaires will also be reviewed by ICF.
Westat will pre-test the revised questionnaires on one case. EPA will approve the revised
questionnaires before they are used in the Survey.
5.2.2 Interviewer Training
The objective of interviewer training is to provide step-by-step guidance to the interviewer
throughout every phase of field Survey interview operations. Interviewer materials and Survey-specific
training will provide the interviewer with in-depth training, a clear set of procedures, and complete
reference documents to use in the course of the study.
5.2.2.1 Summary of Procedures
Interviewers for the CWS Field Survey will be EPA Regional and State employees, and ICF and
Westat employees. All trainees will participate in a one or one and one-half day training session
-------
Section No. 5
Revision No. 0
Date: March 14, 1990
Page 12 of 22
specific to this study. When requested by EPA, trained Westat staff will teach Survey participants in
questionnaire administration. ICF staff will train the participants to conduct well sampling activities. In
specific cases when requested by EPA, trained ICF staff will conduct interviewer training. (The
discussion of well sampling and ICF interviewing training is included in the Quality Assurance Project
Plan for Well Sampling, Data Collection and Processing.)
The Westat Deputy Project Director and Westat Field Director will submit interviewer training
materials to ICF who will prepare standardized training materials (collectively know as the "Training
Manual") from which all trainees will be trained. Interviewer training will consist of lecturing on general
interviewing techniques, specific review of the questionnaires and role playing. General interviewer
training will instruct interviewers on how to avoid biasing the interview, complete a questionnaire,
prob • for additional information, maintain respondent cooperation, respond to questions about
definitions, and clarify imprecise answers.
5.2.2.2 Quality Assurance and Quality Control Procedures
The objective of the quality assurance and quality control procedures for CWS field interviewer
training is to ensure that interviewers are trained to collect information consistently and completely.
The Westat Deputy Project Director will train all Westat trainers. During their first training they will be
observed by experienced Westat staff who will provide them with feedback. The Training Manual, the
standard for all CWS Field Survey interview training, will be used to conduct all interviewer training.
Trainers will instruct from this standard Training Manual to ensure that the same step-by-step
instructions are given to each trainee. Trainers will instruct trainees to rely on their Training Manual,
rather than memory, in the event of uncertainty.
5.2.3 Materials Preparation and Mailout
The objective of Westat's materials preparation and mailout is to prepare packages of interviewing
materials for each CWS sampled well (called a "case") and to send them to ICF in advance of the
commencement of field activities at that CWS. Sampling schedule information will be supplied by ICF.
5.2.3.1 Summary of Procedures
Westat's Data Supervisor, Deputy Project Director, Project Programmer, and Project Systems
Analyst will develop a receipt control (tracking) system to monitor the status of field Survey
questionnaires. This system will be updated upon mailout, or upon receipt of information or
questionnaires by receipt control clerks. Labelled cases will be sent to ICF approximately four weeks
prior to the two week window during which sampling is scheduled to occur according to the sampling
schedule information provided by ICF. Labels will contain the case identification number and the
respondent's name, address and telephone number. Labels containing random numbers to be used
in the selection of wells to be sampled will be placed inside the Team Leader Introduction
-------
Section No. 5
Revision No. 0
Date: March 14, 1990
Page 13 of 22
questionnaire. Bar coded labels with case identification numbers will also be attached to
questionnaires. ICF will forward the interviewing materials to the sampling team along with their Field
Logbook.
5.2.3.2 Quality Assurance and Quality Control Procedures
The objective of the quality assurance and quality control procedure for materials preparation and
mailout is to ensure that all of the information for each case is sent to ICF in advance of the
commencement of field activities according to sampling schedule information provided by ICF. The
questionnaire tracking system will be developed and reviewed to ensure its integrity. Also, any
manually typed information will be accompanied by a computer printout to verify keypunch accuracy.
This printout will be maintained in a logbook.
5.2.4 Monitoring Field Activities and Questionnaire Receipt Control
The objectives of monitoring field activities and questionnaire receipt control are to monitor the
status of interviews and to account for all questionnaires sent out into the field.
5.2.4.1 Summary of Procedures
The monitoring of field activities will be somewhat limited because the state and regional
interviewers are not experienced interviewers under Westat's direct supervision. However, they will be
monitored through their returned questionnaires. All returned questionnaires are reviewed for
problems according to the procedures outlined in Section 5.2.5. Any detected problems that may
have an impact on how survey data are collected in the future will be brought to the Westat Field
Director's attention. The Field Director will then determine whether a call needs to be made to either
the interviewer or field team leader. The Field Director will contact the appropriate person and discuss
the problem, and explain the correct action to be taken in the future. All problems and their
resolutions will be documented in the State Problem Logbook.
Questionnaires for each CWS will be delivered by Westat to ICF, where they will be packed with
the water sampling supplies and sent to the state team leader. All documentation, including
questionnaires and data collected during water sampling will be returned to Westat. Westat will retain
the questionnaires for coding and data entry, and return the water sampling data immediately to ICF.
The computerized receipt control system will keep the following information on each case:
The expected date of water sampling and interviewing at the CWS;
The date on which the case is delivered to ICF;
• The date on which the case is received at Westat after sampling and interviewing;
• A code indicating the status of each finished case (completed, ineligible, etc.); and
• The date on which the water sampling data are delivered to ICF.
-------
Section No. 5
Revision No. 0
Date: March 14, 1990
Page 14 of 22
Information sent by ICF on the expected date of water sampling and interviewing will be received
by Westat and entered into the receipt control system as described in Section 5.2.3.1. Interviewing
packages including questionnaires will be delivered to ICF in three large shipments. Each
questionnaire will be logged out of the system by a Receipt Control Clerk using a light pen to scan
the bar code label on each questionnaire. Both the date and case identification number will be
entered into the system. Returned questionnaires will be logged into Westat's system by scanning the
bar code label. Water sampling data books will be logged out of the system when they are sent to
ICF.
The receipt control system will use the above information to produce reports that will assist the
Field Director in accounting for the survey questionnaires. Three reports will be reviewed regularly to
determine the status of the questionnaires in the field, to indicate when a repeat visit well case needs
to have a well selected, and to indicate the eligibility of those cases returned from the field.
The Overdue Case Report will indicate which cases are overdue from the field by three weeks or
more. Westat will contact the responsible field team leader for an explanation of why the
questionnaire is overdue. These conversations will be documented in the State Problem Logbook. A
copy of the Overdue Case Report shell is shown in Exhibit 5-5.
The Repeat Visit Well Report will indicate those CWSs that have more than one well to be
sampled and that need to have a well randomly selected for an upcoming visit. The field Director will
select for the field team the next well to be sampled using the list of wells collected during the
previous interview. This helps prevent the same well from being sampled more than once and adds
another measure of quality control to the well selection procedure. A copy of the Repeat Visit Well
Report shell is shown in Exhibit 5-6.
The Problem Report will indicate which cases are ineligible, refusals, mismatches (cases sampled
outside of the designated sampling period), and other problems such as updated respondent
identification information, and interviewing problems such as the use of an inappropriate respondent.
A copy of the Problem Report shell is shown in Exhibit 5-7.
5.2.4.2 Quality Assurance and Quality Control Procedures
The objectives of the quality assurance and quality control procedures for monitoring field
activities and questionnaire receipt control are to ensure that the mail-out, interviewing, and return of
questionnaires are performed according to the guideliens specified in Section 5.2.4.1.
All interviewer monitoring problem cases will be documented in a State Problem Log. The
documentation will describe the problem and the action taken to correct it. The field director will
discuss any new or unusual problems with the Deputy Project Director or Project Director to ensure
that correct and consistent interviewing instructions are given to the interviewers.
-------
Section No. 5
Revision No. 0
Date: March 14, 1990
Page 15 of 22
Exhibit 5-5
Overdue Case Report
OVERDUE CASE REPORT
HELU8--
CWJI-
CV9UNE-
REPORT MTE
D*TE Til RECEIVE! C8 RECEIVE* L
3
9
10
11
12
13
II'
IS
It.
17
18
19
20
21
22
73
24
/
/
/
/
/
/
/
/'
/
/
/
/
/
/
/
/
/
/
/
/
/
/
/
/
/
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1 1
1 1
I 1
1 1
1 I
1 I
1 1
1 I
/ 1
1 1
1 1
1 1
1 1
1 1
1 1
I 1
1 1
1 1
1 1
1 1
1 1
1 1
1 1
1 I
1 1
1 1
/ 1
1 1
1 1
I I
1 1
1 1
1 1
1 1
1 1
1 1
1 1
1 1
1 1
1 1
1 I
1 1
t 1
1 1
1 1
1 1
-------
Section No. 5
Revision No. 0
Date: March 14, 1990
Page 16 of 22
Exhibit 5-6
Repeat Visit Report
KEU1D CKSID—- CHSNAI1E-
CUS REPEAT VISIT REPORT
REPORT DATE:
PRESELECTED
RANDOM NUMBERS RANDOM NUMBERS
RAND1 RAND2 RAND3 RAND4 RANDS RANOi RVI RV2 RV3 II HELLS TIRCD— DUEDATE- INELI61BLE
-------
3
9
;0
II
\:
u
14
13
11
17
18
1?
;o
31
Section No 5
Revision No 0
Date. March 14. 1990
Page 17 of 22
Exhibit 5-7
Problem Report
KUID CBII
IMK-
MCI
KM-
tcran
PMK.
coriim MTI o»e
IT
:8
IT
;o
51
37
!J
34
35
14
3T
-------
Section No. 5
Revision No. 0
Date: March 14, 1990
Page 18 of 22
As discussed in Section 5.2.3.2 the tracking system includes checks to ensure its integrity, such
as bar coding to eliminate keypunch errors. In addition to the quality checks already mentioned in the
procedures section, the Overdue Case, Repeat Visit, and Problem Reports will be periodically cross-
checked against ICF's records to ensure their accuracy and completeness.
Westat will also conduct field audits of CWS interviews to observe the implementation of
questionnaire administration by state personnel. Actual procedures used by the field team will be
compared with that taught in the state training session. A report on these audits will be given to the
NFS Quality Assurance Officer, EPA, and ICF.
5.2.5 Initial Editing and Initial Data Retrieval
The objective of the initial editing and data retrieval phase is to identify those Team Leader
Introduction questionnaires with significant interviewing errors, specifically, cases where the identity of
the sampled well is unclear, and to retrieve this missing information from the field team leader.
5.2.5.1 Summary of Procedures
Initial editing will be performed as soon as possible after receipt of the questionnaires in order to
identify the well before the team leader forgets which well was sampled. Data retrieval staff will
telephone the team leaders to obtain the missing information and will document these calls in the
State Problem Logbook.
5.2.5.2 Quality Assurance and Quality Control Procedures
The quality assurance and quality control objective for initial editing and initial data retrieval is to
ensure that any critical interviewer error, especially ones concerning the identity of the well that was
sampled are identified and corrected. The bar coded questionnaires will be processed upon receipt
so that interviewers, or if necessary respondents, can be contacted promptly for data retrieval. In the
event that an interviewer has not recorded information properly, retraining will be conducted during
the data retrieval telephone call to reduce the probability of future errors of the same type. A record
of all problems of this type will be kept in the State Problem Logbook.
5.2.6 Data Preparation and Data Entry
The objective of data preparation and data entry is to create a data file containing the data from
the returned questionnaires. A separate file will be created for each of the three types of
quetionnaires. This is necessary before the information can be properly stored and analyzed.
5.2.6.1 Summary of Procedures
Data preparation and data entry will consist of four steps:
• Coding, which will be a review of questionnaires to resolve any ambiguous responses
and prepare the forms for keypunching;
Data entry, or keypunching of the data;
-------
Section No. 5
Revision No. 0
Date: March 14, 1990
Page 19 of 22
• Editing, which will involve running the keyed data through a set of computerized skip
pattern, range, and logic checks; and
• Data retrieval, where missing information on items designated as key by EPA is
obtained from the CWS by telephone.
Westat will use a coding and data editing system called COED from which a coding manual will
be generated. Coded questionnaires will then be key entered into the appropriate database. The
COED system will also identify data that fail range, logic, and skip checks.
5.2.6.2 Quality Assurance and Quality Control Procedures
The objective of the quality assurance and quality control procedures for data preparation and
data entry is to ensure that data is entered accurately. This objective will be achieved through the
following procedures:
• Training - All coders will be trained by the Coding Supervisor. They will be provided
with coding manuals for their reference. Data retrieval interviewers will be trained by
the Data Retrieval Supervisor. They will also be provided with manuals for their
reference.
• Validating selected cases in coding - The Coding Supervisor will validate 100% of each
coder's work until the supervisor determines that the coder has fully grasped the
procedures, after which at least two percent of the coder's work will be validated.
Maintaining coding 'Problem Logs' - The Coding Supervisor will maintain a log of
problems encountered by coders during the coding process. This log will be
maintained in the coding area and will be immediately available for each coder's
reference. Corrective action strategies will be developed to minimize the reoccurrence
of these problems. Each coder will be briefed on any decisions that affect coding
procedures.
Meeting regularly to discuss problems - At least weekly meetings will be arranged
between the Coding Supervisor and the coders to discuss new coding conventions,
problems, and anything else that impacts the coding process. In addition, the Coding
Supervisor will talk at least weekly with the Data Preparation Supervisor to discuss new
conventions, problems, staffing, and scheduling. These meetings will uncover potential
coding problems and provide for their immediate resolution.
Editing and QA/QC via the COED program - The COED edit program will identify
unacceptable values, missed or inaccurate skip patterns, and simple logical
inconsistencies.
Monitoring of Data Retrieval Interviews - Data retrieval interviewers will be monitored by
the Data Retrieval Supervisor until she is satisfied with their work.
Quality assurance for data entry is achieved by independently double-keying all information to
ensure that it is entered accurately. The second person keying the information is the Verifier and is
responsible for correcting keying errors. Data entry programs are written and checked by the Data
Entry Supervisor.
-------
Section No. 5
Revision No. 0
Date: March 14, 1990
Page 20 of 22
5.2.7 Data Analysis and Reporting
The objectives of data analysis and reporting are to summarize well construction characteristics,
well water treatment, farming practices, pesticide usage, and soil textures in the vicinity of the well,
and other information collected during the CWS Field Survey. This information will be merged with the
well water chemical analyses to determine how pesticide usage is related to ground-water
vulnerability.
5.2.7.1 Summary of Procedures
Missing data frequencies will be reviewed for each questionnaire and the chemical analytes. Data
may be missing due to refusals to participate, chemical analysis quality failures, nonresponse to a
particular questionnaire item, or respondent data failing edit, range, or logic checks. Imputation
procedures will be developed for item missing data and weighting will adjust for unit-level
nonresponse. Weighting will also adjust for the unequal probabilities of selection for different wells
included in the survey.
Population estimates and confidence intervals will be calculated for chemical analyses and many
questionnaire items. Multivariate analytic techniques will be used to examine potential relationships
among the variables. The procedures used for imputation, weighting, and the calculation of
population estimates and confidence intervals will be described in detail in Appendix B of the National
Pesticide Survey Phase I Report. Relational analyses will be described in the NPS Phase II Report.
5.2.7.2 Quality Assurance and Quality Control Procedures
The objectives of the quality assurance and quality control procedures for data reduction,
analysis, and reporting will be to ensure that the data are manipulated and analyzed accurately and
reported clearly. The data will be used in such a manner as to satisfy the objectives of the overall
Survey by illuminating the relationship between pesticide usage, ground-water vulnerability, and
pesticide contamination of drinking water wells. Among the procedures for evaluating the quality of
the questionnaire data will be the following:
• Review the 'Not Answered" and "Don't Know" rates for individual questions. When
combined with feedback from QA audits conducted by the Westat Project Director and
NPS QA Officer this will provide information on the relative utility of responses; and
• "Not Answered" and "Don't Know" rates from the CWS will be compared with those
found from similar questions on the DWS Questionnaire. Differences will reflect both
the relative level of knowledge of CWS owner/operators and well owners, as well as the
impact of using professional, experienced interviewers for the domestic well Survey.
Among the procedures to be used to assure the quality of the imputation, weighting, estimation,
confidence intervals, and relational analyses are the following:
Alternative imputation procedures will be discussed with Westat, ICF, and EPA
personnel. Westat statistical staff will make recommendations on the most appropriate
-------
Section No. 5
Revision No. 0
Date: March 14, 1990
Page 21 of 22
procedures to use. These will then be reviewed by ICF and EPA personnel before
implementation.
After imputation is completed all data will again be passed through edit, range, and
logic checks to ensure the quality of the final data sets. Any imputed data that fail
such checks will be examined and re-imputed if necessary.
The weights must reflect the multiple-stage selection procedure. The distribution of the
weights will be reviewed for each stage in the process by Westat's senior statistical
staff to ensure the accuracy of the programming and to determine if any individual
weight is so large as to be adversely impacting the accuracy of the estimates by
inflating the standard errors. If such a situation arises statisticians will examine the
possibility of weight trimming to reduce the overall mean square error of the estimates.
• Population estimates and confidence intervals will be reviewed by Westat's statistical
staff for consistency. The estimated standard errors will be reviewed to determine if the
implied design effects are within expected levels for such a survey design. Confidence
intervals based upon normal approximations will only be used for those estimates for
which such an assumption will not lead to negative confidence limits.
The potential for temporal effects on chemical results will be examined for those
analytes with a large enough number of contaminated wells. This examination will be
conducted both with and without sampling weights.
5.2.8 Systems Security and Data Backup and Archival
The objective of systems security and data backup and archival is to establish and maintain a
system for the ongoing protection of data.
5.2.8.1 Summary of Procedures
The system and procedures for systems protection and data backup and archival for the NPS will
be the same as those standard for all large-scale Westat data collection efforts. In order to ensure the
security of collected data, Westat will institute the following procedures:
Limit the access to the computer room - Access to the Westat computer room will be
restricted to authorized computer operations personnel.
Secure the VAX/VMS - The VAX/VMS operating system will be password protected to
prevent unauthorized access to the system and all sensitive disk and tape files.
• Backup files - The Master files and all transaction files will be maintained on the Westat
Gaither Microvax cluster with copies of all major components stored off-site at a
commercial tape storage facility. In order to prevent loss of data due to user error or
hardware failure, the computer facility will back-up user files to magnetic tape daily.
Each day's back-up tapes will be retained for more than two months. Lost files can be
recovered within an hour.
• Physically Secure the computer facilities - All computer facilities will be temperature-
and humidity-controlled, protected against fire, power surges, overheating, and flooding
and will be served by a battery-based uninterruptible power system.
-------
Section No. 5
Revision No. 0
Date: March 14, 1990
Page 22 of 22
Archive Periodically - Once a month a full set of back-up tapes containing all user data
will be archived and retained for one full year. Westat uses a remote computer security
facility to store copies of back-up tapes and back-up disk pack media. Data and
analytical results produced as part of the NPS will be archived at significant program
milestones. These milestones include:
Completion of Data Editing;
Completion of Data Analysis; and
Completion of Final Report.
Create Permanent Archive - When data are published, the raw data, methodology, and
procedures will be archived at ICF in order to ensure that the published results are
reproducible.
5.2.8.2 Quality Assurance and Quality Control Procedures
Inherent in the systems security and data backup and archival procedures are the quality
assurance and quality control procedures. Written logs are kept of all back-up activities. The above
procedures are standard Westat practices incorporated into large-scale data collection activities.
-------
Section No. 6
Revision No. 0
Date: March 14, 1990
Page 1 of 23
6. DOMESTIC WELL SURVEY TASKS AND QUALITY ASSURANCE PROCEDURES
The objectives of the Domestic Well Survey (DWS) are to identify a nationally representative
sample of domestic wells, to estimate their frequency and levels of pesticide contamination, and to
gather information needed to relate groundwater vulnerability and pesticide usage associated with
these wells. This information will be gathered through the five questionnaires listed below:
• County Agent Questionnaire;
• Domestic Well Survey Screening Questionnaire;
Domestic Well Questionnaire;
Domestic Well Team Leader Introduction and Well Observation Record; and
Domestic Well Local Area Expert Questionnaire.
EPA has no comprehensive listing of domestic water wells which serves as a sampling frame.
Therefore, the Agency used a sampling methodology to randomly select domestic water wells with an
emphasis on geographic areas that have high pesticide usage and ground-water vulnerability. EPA
stratified the more than 3,000 counties in the United States by an indicator of pesticide usage and
ground-water vulnerability (known as First-Stage Stratification) (see Exhibit 6-1) and then randomly
selected 90 counties, oversampling counties that are in the stratum of high pesticide usage and/or
high ground-water vulnerability. This process created the First-Stage Sample.
Exhibit 6-1
Stratum
1
2
3
4
5
6
7
8
9
10
11
12
Pesticide Use
High
High
High
Medium
Medium
Medium
Low
Low
Low
Uncommon
Uncommon
Uncommon
Groundwater Vulnerability
High
Medium
Low
High
Medium
Low
High
Medium
Low
High
Medium
Low
-------
Section No. 6
Revision No. 0
Date: March 14, 1990
Page 2 of 23
ICF is responsible for subdividing each of the 90 selected counties into areas of high ground-
water vulnerability and cropping (known as Second-Stage Stratification). This stratification is based on
information collected through in-person interviews with local county agricultural cooperative extension
agents (i.e. administration of the County Agent Questionnaire).
Westat will conduct a Random Digit Dial (ROD) screening interview of households in each of 84
counties (wells were previously identified in six counties as part of the NFS Pilot Survey). The
screening interview will be used to select a sample of households (known as Third-Stage sample) that
have a drinking water well and are eligible for the full scale Survey. After selecting a sample of
households, a field team consisting of an ICF Team Leader and a Westat interviewer will visit the
household. The Team Leader will collect well water samples and complete the Well Observation
Record. The Well Observation Record will be used to record information about the well and the
surrounding area. The Westat interviewer will administer the DWS Questionnaire to homeowners and
residents to gather information on the use of the well water and the construction of the well. To obtain
information on the land use of the area surrounding the wells being sampled (e.g., presence of
landfills, wastewater treatment facilities, farming, etc.), the Westat interviewer will also conduct an
interview with a local area expert (e.g., the County Agricultural Cooperative Extension Agent).
6.1 Second-Stage County Agent Interview
The primary objective of the County Agent Questionnaire is to gather cropping and pesticide
usage information for each of the counties in the DWS Field Survey. This information will be used to
determine the oversampling areas ("cropped and vulnerable") within each of the selected counties.
6.1.1 Summary of Procedures
The County Agent Questionnaire will be administered as an in-person interview with the County
Agricultural Cooperative Extension Agent. The information obtained with the County Agent
Questionnaire will be used to establish the "Cropping Category" for the areas in each United States
Geological Survey (USGS) quadrangle map in each of the selected counties. The Cropping Category
will be defined by the percent of land area farmed (i.e. more than 50%, 25 to 50% or 0 to 25%) and
the pesticide usage (i.e. higher, lower or about the same) in each quadrangle relative to the rest of the
county. This cropping information will be overlaid with the hydrogeologic ground-water vulnerability
information (i.e. DRASTIC scores) for each county to define the areas for oversampling. The resulting
Second-Stage County Cropped and Vulnerable Maps will be used for oversampling during the DWS
Screening Survey (Section 6.2.) performed by Westat. (These mapping activities are described in
further detail in the Appendix A of the Quality Assurance Project Plan for Hydrogeologic
Characterization and Second-Stage Stratification Activities.)
-------
Section No. 6
Revision No. 0
Date: March 14, 1990
Page 3 of 23
All County Agent Questionnaires will be administered by four ICF staff members. These
interviewers will receive interviewer training specific to the County Agent Questionnaire. A Question-
by-Question Specification Manual will be prepared to ensure that the interviewers administer the
questionnaires consistently. (The County Agent Questionnaire administration activities are described
in further detail in Appendix B of the Quality Assurance Project Plan for Hydrogeologic
Characterization and Second-Stage Stratification Activities.)
Westat will review the County Agent Questionnaire to ensure that the language and format of the
questionnaire is consistent across all interviews and conforms to appropriate questionnaire design.
Westat will review the questionnaire to reduce ambiguity and response errors.
The completed County Agent Questionnaires will be reviewed by other project staff when they are
returned from the field at which point initial data editing and data retrieval will be conducted.
6.2 DWS Screening Survey
The objective of the DWS Screening Survey is to identify in each county a sample of eligible
households possessing drinking water wells. The number of sampled eligible households in each
county will vary according to the stratum designation.
6.2.1 Questionnaire Preparation
The objective of questionnaire preparation is to produce a questionnaire that collects the
information specified by EPA. The DWS Screening Questionnaire will be a Computer Assisted
Telephone-Interview (CATI) and will be used to identify and select households eligible for DWS
sampling and data collection.
6.2.1.1 Summary of Procedures
Westat will develop the DWS Screening Questionnaire by reviewing and reformatting the
questionnaire. The DWS Screening Questionnaire will be administered as a CATI questionnaire which
contains approximately 30 questions. Westat will program the questionnaire for CATI and train
telephone interviewers to administer the screening questionnaire. All responses obtained from the
screening Survey are entered into a database by the program. Data range and skip pattern checks
are performed during the interview.
6.2.1.2 Quality Assurance and Quality Control Procedures
Westat's quality assurance and quality control goals for the DWS Screening Questionnaire are to
ensure that the information obtained is consistent across all interviews and to determine if the eligible
well is in a 'cropped and vulnerable" portion of the county according to information provided by ICF.
The quality assurance and quality control procedures also ensure that the questionnaire collects the
information from the respondent needed to determine domestic well eligibility according to criteria
established by the EPA. These goals will be achieved through extensive in-house reviews of the
-------
Section No. 6
Revision No. 0
Date: March 14, 1990
Page 4 of 23
questionnaire by experienced Westat staff. The questionnaire will be reviewed by ICF and EPA prior
to its use.
6.2.2 Preparation of County-specific Questions
The objective of developing county-specific questions is to produce a list of questions specific to
each county which will be needed to identify the sample of households for which to administer the
DWS Screening Survey. This will be accomplished by identifying rural areas of each of the selected
counties in which to conduct the DWS Screening Survey, and "cropped and vulnerable" regions to be
oversampled. For this Survey, EPA is not interested in urban regions of counties and these regions
must therefore be identified and eliminated from potential inclusion in the Survey.
6.2.2.1 Summary of Procedures
Preparing county-specific questions has three major steps: cluster specifications; map
preparation; and developing, programming, and testing the questions. Cluster specification staff
develop county specifications according to the procedures described in the series of memos included
as Appendix C. An example of a Domestic Well Screening Survey Cluster Identification form is shown
as Exhibit 6-2. Most of the codes on this form are described in the initial document in Appendix C.
The group number refers to a Westat internal grouping of the counties into sets of size four. Adjusted
C1 is the initial number of clusters, adjusted for percent urban. C2 is the number of clusters initially
sent to the telephone center and is an adjustment to ensure that the numbers of clusters exceeds the
desired number of completes and thereby reduce the number of completes per cluster.
C2=maximum{adj C1, nx1.25} (for early counties, before adjustments were made for urbanicity,
C2=maximum{C1, nx1,25}). C3 is the larger number of clusters to be identified to provide a source
of reserve clusters if needed. This form is sent to the Cluster Identification Supervisor for selection of
clusters.
Mapping staff will begin by collecting local county maps for each upcoming county. They will use
these local county maps to draw urban areas and major landmarks onto the Second-Stage County
Cropped and Vulnerable Maps provided by ICF. The maps provided by ICF will already indicate the
target areas to be oversampled. Mapping staff will also check the zip code boundaries on the
Second-Stage County Cropped and Vulnerable Maps against those on a Rand McNally Zip Code
Atlas. (For some counties, the second-stage maps will not include zip code boundaries, in these
cases boundaries will be drawn from the Rand McNally atlas.) Discrepancies in zip code boundaries
will be resolved by contacting the local county post office. Since EPA is not interested in urban
regions for this study, a list of known urban areas to be excluded from each county will be prepared
from Census Bureau data.
-------
Section No. 6
Revision No. 0
Date: March 14, 1990
Page 5 of 23
Exhibit 6-2
Domestic Well Screening Survey Cluster Identification Form
OORESTIC KEll ECREEKINB SURVEY :ate: ^ / 73/ ^
— T—— -*»^w -..*.
CLUSTER IDENTIFICATION
Prefixes in the non-oversa»oled area?
n = target nuaoer of c:ioletes
: : Darcsn: ST housenoiss ;.•• csuntv m:n *eils ;• ,'es, i;st:
t - percent of housenolds in oversaaoled area
' r percent of households only in the
r.snoversaaoted area
r - response rate
s = 5i;e of screening saagle*
:! - nuaoer of clusters generated froa equation
c2 - adjusted nuaOer of clusters for cluster specs*
:3 = adjusted nuaoer of clusters for additional cluster id*
n(cluster) = nuaber cf phone nuabers per cluster* Prefixes in uroan areas? f^N
*Note: s, :2, ind n(cluster) are the cluster ID specs If ves, exclude:
n «/ ?<*
Screening teqins: ///?» ° ^
Screening ends: // /2^/ ^ 4.€L?A.?2
STRflTUN COUNTY 6ROUP
IDI NAflŁ STATE * » P, t v i. ci - -?J -t cM rlc'u.
I'f0n* IL 1? M °'31 <>•'» ».15 3-75 35.69 12.13 0.22 S5.97 SS.97 83.W
-------
Section No. 6
Revision No. 0
Date: March 14, 1990
Page 6 of 23
Mapping staff make the initial draft questions to determine if a household is located within the
"city limits" of an urban area to be excluded. For example, "Do you live within the city limits of
Richmond?" If the boundary of the urbanized area is unclear, this boundary will be overestimated to
adjust for probable growth since the 1980 census. Once all urban and urbanized regions in the
county have been excluded, the zip codes in the remaining "target areas" will be compiled and the
physical boundaries of the target area will be defined by using landmarks (i.e. rivers, mountains,
highways, etc.). These landmarks will then be used in a series of questions to determine well location
relative to the target area. Final decisions on "smoothing" the boundaries will be made by the DWS
Mapping and Cluster Specification Supervisor (see Appendix D for additional details). After approval,
the county-specific questions are submitted for CATI programming and testing. The County Questions
Development Supervisor will log onto the VAX to test the program using multiple mock interviews.
6.2.2.2 Quality Assurance and Quality Control Procedures
The objective of the quality assurance and quality control procedures for preparing county-
specific questions is to ensure that zip code boundaries and urban regions are accurately identified
and that these identifications are made consistently across all counties. In order to ensure the
accuracy of these procedures, the mapping staff will be trained specifically for performing the research
to confirm zip code boundaries and city limits for the Second-Stage County Cropped and Vulnerable
Maps. To ensure the consistency of these identifications across all counties, all final decisions
regarding the smoothing of boundaries will be made by the DWS Mapping and Cluster Specification
Supervisor. Specific procedures for quality assurance include:
Reviewing (by the ICF Project Director) the overall procedure for determining clusters
(the first letter included in Appendix C);
• Reviewing (by the Cluster Specifications Supervisor and Project Director) the Domestic
Well Screening Survey Cluster Identification Form for each county;
Maintaining a control chart on county-by-county completion rates in the telephone
research center (Exhibit 6-3). This chart will be reviewed by the Westat Project
Director; if the chart indicates the need for modifying cluster specification procedures
the Project Manager will institute such corrective action;
Preparing a Zip Code Check List for each county (Exhibit 6-4) that will be used to
document all zip codes identified by the telephone center, ICF, or the Rand McNally Zip
Code Atlas. This check list will identify whether a zip code includes the target area and
what percent of it is in an urban area. This form will be reviewed by the Mapping
Supervisor;
• Reviewing (by the County Questions Development Supervisor) county-specific questions;
and
Testing (by the County Questions Development Supervisor) the CATI programming.
-------
Section No. 6
Revision No 0
Date: March 14. 1990
Page 7 of 23
Exhibit 6-3
Control Chart for Attribute Data.
CONTROL CHART FOR ATTRIBUTE DATA
9-24
ACC052886
-------
Section No. 6
Revision No. 0
Date: March 14, 1990
Page 8 of 23
Exhibit 6-4
Zip Code Checklist
EPA OOHESTIC WELLS •918312
NEW LONDON. CT
ZIP CODES
-HIGH COVER
PREFIXES
DIALED IN
:NA«EJ COUNTY
ZIP CODES
INDENTIFIED
ON HAPPING
RESOURCES •>
YES
ZIP COOES
INCLUDES
TARGET
(OVERSAHPLED)
AREA
0.
YES
NO
ZIP CODES
ALL IN
URBAN
AREA ?
YES
NO
0. -HAT X OF LAND IN COUNTY IS THE TARGET/OVERSAHPLED
AREA ? .5 X
-------
Section No. 6
Revision No. 0
Date: March 14, 1990
Page 9 of 23
6.2.3 Selection of Households to be Screened
The objective of this task is to identify the subset of households for which to conduct the DWS
Screener Survey.
6.2.3.1 Summary of Procedures
The telephone research center (TRC) has a list of all telephone prefixes (the six digit combination
of area code and exchange) being used in each county. Each prefix is assigned to one of three
categories:
1. Only used in urban or urbanized areas whose residents are by definition ineligible for
this Survey;
2. Only used in areas that are to be undersampled (non-target) in the NPS. These
prefixes may also include some ineligible areas; or,
3. Used (entirely or partially) in areas that are oversampled.
Prefixes in the first category are excluded from the screener sample. Each of the other prefixes is
divided into 100 clusters of 100 numbers each. An eight-digit cluster is defined by appending one of
the numbers 00 through 99 to the six-digit prefix. The number of clusters to be screened for cluster
identification is taken from the cluster specifications described in Section 6.2.2. This number of
clusters is selected systematically from the list of eligible clusters. Clusters are selected in category 2
at one-third the rate of category 3. (Undersampling of respondents in category 3 clusters who are in
non-target areas is conducted automatically as part of the CATI programming. If there are no prefixes
in category 3, i.e., there are no oversampled areas in the county, then no undersampling of clusters is
done.)
Clusters of telephone numbers tend to be released primarily for residential or for commercial use.
The TRC will call one telephone number within a cluster to determine whether the telephone number
is for a residence or for a commercial establishment and whether they are located in the sampled
county. Each number will be tried up to 7 times to determine eligibility. These calls will be made up
to a maximum of 3 evening, 2 per daytime, and 2 per weekend calls. This will ensure that eligible
wells are not identified as a function of time of day that respondents are at home.
6.2.3.2 Quality Assurance and Quality Control Procedures
The objective of the quality assurance and quality control procedures for selection of households
to be screened is to ensure that the correct number of clusters are selected per county. The set of all
zip codes in the county for which prefixes are identified wili be reviewed by the ROD Screener
Supervisor and compared against the maps developed for determining county-specific questions.
When additional zip codes are identified on the maps the ROD Screener Supervisor will notify the
Cluster Identification Supervisor of the need to check the information from adjoining counties.
-------
Section No. 6
Revision No. 0
Date: March 14, 1990
Page 10 of 23
The cluster specifications will be reviewed for reasonableness by the Project Director when they
are sent to the Cluster Identification Supervisor. Meetings will be held between these staff members
to review the sampling procedure used for the initial counties. Progress on identifying clusters will be
reviewed daily by the ROD Screener Supervisor. If questions arise concerning the need for additional
clusters they will be discussed with the Deputy Project Director and Project Director. The telephone
interviewing supervisors will monitor a sample of cluster identification calls as part of their regular
monitoring of interviews as described in Section 6.2.4.
6.2.4 Questionnaire Administration
The objective of questionnaire administration is to screen the sample of households identified
above for eligibility in the DWS Field Survey. The DWS Screener Questionnaire will be administered
as a telephone interview by Westat's TRC using CATI. Up to 7 calls to each telephone number will be
made before a case is "closed out."
Responses to the DWS Screener Survey will be directly entered into a database. Daily reports on
the number of completed interviews, refusals, and ineligibles will be generated from the database.
These reports will be used to monitor the daily progress of the screening Survey. Production reports
which compute the number of hours worked, the number of cases completed, and the average time
taken to complete a case will be generated.
6.2.4.1 Summary of Procedures
Westat uses a proprietary computer program to conduct CATI Surveys. This program will
automatically maintain records of the calls and results.
The interviewer will enter one telephone number at a time into the CATI system. The system will
then guide the interviewer through the interview by automatically following skip patterns according to
the responses entered by the interviewer. When the interview is completed, the CATI system will
assign a disposition code indicating the status of the case. The Call Record will then be sent back to
the main TRC building for processing.
An output known as the Respondent Information Sheet (RIS) will be generated from the database.
The Respondent Information Sheet contains the respondent's identification number, name, address,
and telephone number.
Interviewer Training will be conducted to provide step-by-step guidance to the interviewer
throughout every phase of screening Survey interview and CATI operations. Interviewer materials and
Survey-specific training will provide the interviewer with a clear set of procedures and complete
reference documents to use during the telephone Survey.
Westat will be responsible for conducting training specific to this Survey interview and developing
and reproducing the interviewer training materials. Interviewer training will consist of a two day
-------
Section No. 6
Revision No. 0
Date: March 14, 1990
Page 11 of 23
training session specific to the DWS Screening Survey. Westat will prepare a DWS Screening Survey
Training Manual for interviewer training (Appendix E).
Interviewer training for the DWS Screening Survey will include:
• Introduction to the purpose and content of the DWS Screening Survey;
• Question-by-question review of the Survey questionnaire;
• Review of the Interviewer Training Manual;
Interviewer role playing; and
• Question and answer sessions to review Survey procedures.
6.2.4.2 Quality Assurance and Quality Control Procedures
The objective of the quality assurance and quality control procedures for administration of the
DWS Screening Questionnaire is to ensure that the information obtained by administering the
screening questionnaire is consistent across all counties. Inherent in the programming will be the
following quality assurance and quality control procedures:
Skip pattern logic of the CATI questionnaire is fully computerized so that interviewer
choice in question branching is eliminated;
• The wording of questions appropriate to the respondent's county is automatically
performed for the interviewer by the CATI software;
Validation of response codes for closed-ended questions is performed by the CATI
software during the interview so that invalid codes cannot be entered into the data files;
• Internal logic checks are pre-programmed for on-line editing of the questionnaire as it
is being administered; and
• Consistency between certain related interview items is automatically checked by the
CATI software. The system will instruct the interviewer to re-ask the question or probe
for additional information in the event that inconsistent entries are detected. This
minimizes respondent error and interviewer entry error.
The following additional procedures will also be followed to ensure the quality of the actual DWS
Screening Survey interview:
Monitoring of interviewers by supervisory and project staff;
• Maintaining telephone supervisor problem logs;
Producing daily and/or weekly production reports; and
• Conducting frequent meetings between telephone interviewers, programmers,
supervisors, and project staff during which problems will be reviewed.
-------
Section No. 6
Revision No. 0
Date: March 14, 1990
Page 12 of 23
Coding and editing of screening data will be performed by the CATI computer programs.
Additional procedures include daily and weekly reports generated and reviewed by the management
staff.
The objective of monitoring the screening interviewing activities is to ensure that the questionnaire
is administered consistently across all interviews. CATI Interviewing Supervisors will conduct interview
monitoring. At least five percent of the interviews will be monitored. The interviewers will not know
when they are being monitored. A debriefing session will be held with the interviewers to identify any
problems that have occurred during the screening interview process. A copy of the interviewing
monitoring form completed by the supervisor is shown as Exhibit 5-2.
6.2.5 Selection of Field Sample
The objective of the selection of the field sample is to identify a sample of DWSs that meet the
Survey eligibility criteria and represent DWS wells in the 48 continental states of the U.S. Westat is
responsible for developing, reviewing, and implementing the sampling plan. This will be done by
compiling the DWS Screening Survey results to determine whether the required number of eligible
respondents have been contacted for each county.
At the end of each DWS Screening Survey, the interviewer will set up a tentative time for the DWS
Field Team to interview the respondent. All interviews in a county are to be completed during a 2
week period. A calendar for this period and a map of the county will be posted on an easel in the
CATI area during the telephone interviews. Interviewers will attempt to make these appointments
during the first week of the field period. Respondents will be told that these appointments are just
tentative and that they will be contacted by telephone prior to the scheduled appointment to confirm
the date and other aspects of the operation.
A minimum of three hours will be scheduled between appointments. After making the
appointment and completing the interview, the interviewer will fill in the time period on the calendar
that has just been assigned to prevent the assignment of the same time slot to more than one
respondent.
All relevant information from the interview will be summarized on a Respondent Information Sheet
(RIS) and sent to ICF well in advance of the commencement of field activities for that DWS.
6.2.5.1 Summary of Procedures
The DWS Screening Survey is scheduled to take roughly three weeks for each county.
Throughout the three week screening period, the DWS Screening Survey Manager will review the
DWS Screening Survey results to determine whether the required number of eligible respondents have
been identified in the county. One of two conditions will exist as shown below:
-------
Section No. 6
Revision No. 0
Date: March 14, 1990
Page 13 of 23
Based upon initial results for the county, it is anticipated that completing the original
set of selected telephone numbers (number of clusters multiplied by the amount of
telephone numbers per cluster) will provide at least the required number of sampled
households. All remaining cases without final status codes will continue to be
processed as planned, with final close-out of the county three weeks after work
commenced.
Based upon initial results for the county, it is not anticipated that completing the
original set of selected telephone numbers will provide the required number of sampled
households. It will then be necessary to begin work on additional telephone numbers.
This will either be accomplished by increasing the amount of telephone numbers per
cluster or by increasing the number of clusters. This decision will be made by the
Deputy Project Manager, in consultation with the Project Manager as needed. The
general procedure will be as follows: if no more than two sampled households are from
any single cluster the amount of numbers per cluster will be increased; if some clusters
already contain at least three sampled households the number of clusters will be
increased (using the reserve clusters identified earlier, see Section 6.2.2). If such an
increase would cause the total amount of telephone numbers called in a county to
exceed 1,000, the Westat Project Director will review the situation with the ICF Project
Director to examine the cost/benefit trade-offs.
When a county is closed-out in the telephone center the DWS Screener Survey Manager will
review the number of sampled households that have been selected. (To be a sampled household a
case must either be coded as a complete (C) or a partial complete missing only farmer information
(PCF).) There are three possible situations as shown below:
The number of sampled households is equal to the number that was desired. The
respondent information sheets will be printed and sent to ICF by the Screener Survey
Manager.
The number of sampled households is less than the number that was desired. This
can only happen when an excessive number of telephone calls are required and the
ICF Project Director has given permission to close-out the county. The RISs will be
printed and sent to ICF by the Screener Survey Manager.
• The number of sampled households is greater than the number that was desired. The
sampled households must be subsampled to achieve the desired total. The procedure
used for systematic subsampling is given below. The RISs for the selected subsample
will be printed and sent to ICF by the Screener Survey Manager.
The method of systematic random subsampling will be as follows:
Given:
C = Number of complete, eligible respondents
R = Required number of complete, eligible respondents
C > R
n = number complete, eligible respondents to be dropped
I = skip interval
-------
Section No. 6
Revision No. 0
Date: March 14, 1990
Page 14 ot 23
Sort C by oversampled/undersampled.
n = C-R
I = C/n
Select a random number, RN, where 0 <. RN <. I.
Compute the list RN, RN + I, RN+2I,..., RN + (n-1)l.
The respondents to be dropped are those on the sorted list corresponding to the numbers just
computed, rounded up to the nearest integer.
6.2.5.2 Quality Assurance and Quality Control Procedures
The objective of the quality assurance and quality control procedures for the selection of the field
sample is to ensure that the procedures for identifying the sample of DWSs for inclusion in the DWS
Field Survey are executed accurately and consistently across all counties. All final decisions
regarding the selection of the field sample will be made by the DWS Screening Survey Manager. This
will ensure the consistency of the selection process. The systematic random subsampling method
ensures accuracy and consistency in the selection of the field sample in the event that more than the
required number of eligible respondents is identified.
6.3 Domestic Well Field Survey (DWS)
The purpose of the DWS Field Survey is to collect and analyze information pertaining to the
domestic wells from which water samples are drawn. This information is collected through the
administration of three "instruments," the DWS Questionnaire, the DWS Team Leader Introduction and
Well Observation Record, and the Local Area DWS Questionnaire. Examples of these instruments
appear in Appendix F.
6.3.1 Questionnaire Preparation
The objective of questionnaire preparation is to ensure that the instruments used in the DWS
Field Survey are worded and formatted such that they collect the information specified by EPA to
satisfy the objectives of the DWS Field Survey. Specifically, the information that is gathered includes,
but is not limited to, depth of the well, well construction characteristics, water treatment, depth to
groundwater, general aquifer characteristics, soil texture in the immediate vicinity of the well, and
pesticide usage in the vicinity of the well. Westat will modify the Survey instruments developed from
the pilot phase of the NPS and incorporate ICF's and EPA's comments.
6.3.1.1 Summary of Procedures
The DWS Questionnaire and DWS Team Leader Introduction and Well Observation Record will be
completed in the field at the time of well water sampling. The respondent for the first part of the DWS
Questionnaire and the DWS Team Leader Introduction will be the head of household or the spouse of
the head of household for the residence contacted during the screening Survey. The DWS
-------
Section No. 6
Revision No. 0
Date: March 14, 1990
Page 15 of 23
Questionnaire will be designed to collect information regarding water usage, water treatment, well
construction, pesticide use and farming practices. The section on farming practices will be asked of
all farmers of the well property in the last 5 years. Well construction information will be asked of the
well owner. These may or may not be the head of household or spouse of the head of household in
the selected residence. The DWS Team Leader Introduction portion of the DWS Team Leader
Introduction and Well Observation Record will introduce the respondent to the Survey by presenting
an overview of the Survey objectives.
The DWS Well Observation Record portion of the DWS Team Leader Introduction and Well
Observation Record will be completed by field team members during well water sample collection
activities. This instrument will be used by the field team to record information regarding soil
characteristics and other significant features in the vicinity of the well.
The Local Area DWS Questionnaire will be administered to a person familiar with local conditions
and usage of land surrounding the DWS being sampled. This person will usually be the County
Agricultural Extension Agent. This interview should be conducted within one week of the well water
sampling activities.
All completed instruments will be sent to Westat for data preparation. Once the records are
received, they will be entered into a receipt control file.
6.3.1.2 Quality Assurance and Quality Control Procedures
The original DWS pilot questionnaires were prepared and pretested by EPA and supplied to
Westat. The questionnaires were reworded and reformatted by Westat, and reviewed by in-house
experts as well as by ICF and EPA. EPA will review the revised questionnaires before they are used in
the Survey.
6.3.2 Questionnaire Administration
The objectives of questionnaire administration are to introduce the respondent to the Survey and
collect information regarding water usage, water treatment, well construction, farming practices, and
pesticide usage.
6.3.2.1 Summary of Procedures
The DWS Questionnaire will be administered to collect the information regarding water usage,
water treatment, well construction, farming practices, and pesticide usage. The Team Leader
Introduction portion of the Team Leader Introduction and Well Observation Record will be
administered by ICF personnel to introduce the respondent to the Survey by presenting an overview
of Survey objectives. (The member(s) of the field team responsible for collecting the water samples
will complete the Well Observation Record portion of the Team Leader Introduction and Well
Observation Record.) Trained Westat personnel will administer the DWS Questionnaire.
-------
Section No. 6
Revision No. 0
Date: March 14, 1990
Page 16 of 23
Appointments will be made for interviews with household residents, well owners, and farmers prior
to the field visit. The initial respondent will be the head of the household or their spouse. If additional
farmers or well owners are discovered during the interview the Westat interviewer will contact that
person and arrange to conduct the interview at a convenient time during the two-week sampling
period. Whenever an interview is being conducted without concurrent well sampling the Westat
interviewer will introduce the respondent to the Survey. Upon completion of the DWS Questionnaire
the interviewer will assist with water sampling activities. (For more details on water sampling see the
Well Sampling, Data Collection, and Processing Quality Assurance Project Plan.)
The interviewer will contact the local county agent to confirm the appointment scheduled by the
DWS Field Director. Before conducting the county agent interview the interviewer will drive to all wells
that still remain to be sampled to accurately identify their location. This information will then be used
to describe the locations in question to the county agent, while retaining the maximum level of
respondent confidentiality possible.
Upon arrival at the well site, the Field Team Leader will locate the respondent and introduce the
respondent to the other field team member(s). The Westat employee will seek a convenient location
in which to administer the questionnaire while the other field team member(s) begin the water sample
collection procedures. The Westat employee will ask the questions in the questionnaire following the
appropriate skip patterns. The responses will be recorded in pencil directly in the questionnaire.
Upon completion of the questionnaire, the interviewer will answer any questions the respondent may
have regarding the Survey.
6.3.2.2 Quality Assurance and Quality Control Procedures
The objective of the quality assurance and quality control procedures for questionnaire
administration is to ensure that the information collected through administration of the questionnaire is
recorded accurately and that the questionnaire administration is performed consistently across all
counties. Westat will employ the following procedures to ensure the quality of the questionnaire
administration:
Interviewers will receive one-and-one-half days of Survey-specific training given by the
Deputy Project Director and other project staff. The training will include question-by-
question discussion of the questionnaires and role playing. Each interviewer will be
provided with a manual to assist with training and to serve as a reference during
interviews.
• Interviewers will be observed by the Deputy Project Director and other project staff
during the training role plays. Interviewers showing difficulty in understanding or
administering the questionnaires will be given additional training until they demonstrate
an acceptable skill level. In extreme cases, interviewers may be dismissed from the
study.
-------
Section No. 6
Revision No. 0
Date: March 14, 1990
Page 17 of 23
• Interviewers will review all questionnaires before leaving an interview site to insure
completeness of all responses.
• Interviewers will make marginal comments that can be further evaluated at a later time
when additional information is offered that may be relevant to the Survey.
• Interviewers will conduct an initial edit of all questionnaires before returning them to the
DWS Field Supervisor.
• Interviewers will remain in frequent contact with the DWS Field Supervisor as needed.
This will typically result in telephone contact every other day.
• In-field quality audits will be conducted by the NPS QA Officer and Westat supervisory
staff. This will include the Project Director, Deputy Project Director, and the DWS Field
Supervisor. Westat auditors will use the Field Observation Form (Exhibit 6-5). This
form includes a check list on general interviewing techniques, a listing of strengths and
weaknesses of the interviewer's performance, and summarizes the review discussed
with the interviewer.
6.3.3 Questionnaire Receipt Control
The objective of questionnaire receipt control is to identify questionnaires that are not returned
from the field in a timely manner so that the progress of the field work can be accurately measured.
6.3.3.1 Summary of Procedures
The receipt control of the DWS questionnaires from the field is based upon the identification
labels attached to each individual questionnaire. The identification number is of the form xx-xx-xxxx-x.
The first two digits refer to the first-stage stratum to which the county belongs (01 - 12); the second
two digits are a county identification number corresponding to the order in which they are surveyed
(00 - 84); the next four digits are the unique identifier for that well, attached to the case as part of the
DWS Screener Survey; the final digit identifies the type of questionnaire (1 for the DWS main
questionnaire/local area questionnaire, and 2 for the Team Leader Introduction and Well Observation
Record). Labels with the appropriate 9-digit identification will be attached to the front of each
questionnaire by the DWS Field Supervisor before they are mailed to the field team.
Questionnaires will be returned by the interviewer to the Field Supervisor after completion of
questionnaire administration in a county. The ICF Team Leader will have the option of giving the
Team Leader Introduction and Well Observation Records from an entire county to the Westat
interviewer for mailing or mailing them to the Westat Field Supervisor directly.
Tracking of returned DWS questionnaires within Westat will be automated using a system
analogous to that described for the CWS in Section 5.2.5. Mail-out and receipt information will be
entered into the system by scanning the bar code label on the front of each questionnaire. The result
code for each section of the questionnaires will be key entered into the system to provide information
on the completeness of each section.
-------
Section No. 6
Revision No. 0
Date: March 14, 1990
Page 18 of 23
Exhibit 6-5
Field Observation Form
Cbwrvwi
UUi
401
brnmi
1.
•J.
•till
Uw
t« 9«t thi.
1 1 1 I 1
1 1 1 1 1
1 1
1 1 1
1 1 1 1 1
1 I
1 1 1
1 1 1 1 1
If
U*t b«i«* U
i. Stranq«t point «i
thl*
JC.
•r*«l«n •tut u« UiUrviiMr (
t, ttt.)i
;•*•* pcinta 4U
i .
-------
Section No. 6
Revision No. 0
Date: March 14, 1990
Page 19 of 23
6.3.3.2 Quality Assurance and Quality Control Procedures
The objectives of the quality assurance and quality control procedures for questionnaire receipt
control are to ensure the knowledge of the status of all questionnaires. When questionnaires are
initially sent out to the field the DWS Field Supervisor will complete a DWS Transmittal Sheet
documenting the contents of the package, along with the date and names of who sent it and to whom
it is addressed. Exhibits 6-6 and 6-7 are examples of Transmittal Sheets used to send forms
separately to the Westat interviewer and ICF Team Leader, respectively. When the questionnaires are
returned to Westat they must be accompanied by similar transmittal sheets.
The Field Supervisor then indicates the status of each questionnaire on the Receipt Control Form
(Exhibit 6-8). For the main questionnaire it is necessary to indicate the status separately for each
section of the questionnaire. This form is then used for the key entry described in Section 6.3.3.1.
6.3.4 Initial Editing and Data Retrieval
The objective of initial editing and data retrieval is to ensure completeness of all questionnaires
and to collect missing information in a timely method.
6.3.4.1 Summary of Procedures
Upon receipt of all questionnaires from the field, the DWS Field Supervisor will review the
questionnaires for completeness. The supervisor will review all marginal comments for clarity.
Whenever missing information is discovered the supervisor will call the field interviewer and clarify the
items in question.
6.3.4.2 Quality Assurance and Quality Control Procedures
The Deputy Project Director and Project Director will be regularly informed of the status of in-
coming questionnaires by the Field Supervisor. When a questionnaire reveals a situation with which
the Field Supervisor is not familiar and which is not explicitly discussed in the training manual the
supervisor will refer the situation to either the Deputy Project Director or the Project Director. If
necessary, they will consult with the ICF Project Director or his designee for further clarification.
6.3.5 Data Preparation and Data Entry
The objective of data preparation and data entry is to create a data file containing the data from
the returned questionnaires. A separate file will be created for each of the three types of
questionnaires. This is necessary before the information can be properly stored and analyzed.
6.3.5.1 Summary of Procedures
Westat will use the COED system described in Section 5.2.6. for Data Preparation. The
procedures and quality assurance afid quality control measures for data preparation and data entry
for the domestic well survey are exactly the same as those used for the CWS survey detailed in
Section 5.2.6.
-------
Section No. 6
Revision No. 0
Date: March 14, 1990
Page 20 of 23
Exhibit 6-6
Westat Interviewer Transmittal Sheet
DWSThranM
FROM: O-
day
A/
Nairn of Parson aandfcg Mormaoan/cfty,
TO:
ITVQ
Q^ CounryfcUo
OWS
BooM«:
G2-
03"
Sampling SctwdU*
IntwvtMMr Pacfcag* Chaeic
Taam Laadar Pacfcaga Chadc
Othar.
WadObsarvatkm
Raeore:
D
D
D
D
D
D
D
D
D
'D
D
D
lONumbar
D-on-DD-nnnn
paga page
-------
Section No 6
Revision No. 0
Date: March 14, 1990
Page 21 of 23
Exhibit 6-7
ICF Team Leader Transmittal Sheet
OatK
FROM:
Nairn of Panon aanotog Wo
70; /\ >(L^ -Zot- TT/A/
Nam of Pi
Me
Irton
ows
D
D
D
D
D
D
n
D
D
D
n
D
__ County Mao
Irtarvtmw Package Check
Team uadar Padeaga Qwcfc
Othar.
Wai Obsatvaflon
RaooRC
a
lONumtwr
D-
-E3E30E
•E3E1IZ3E3
0-G3E3-13E'
17"
page P>g«
-------
Section No. 6
Revision No. 0
Date: March 14, 1990
Page 22 of 23
Exhibit 6-8
Example of a Completed Receipt Control Form
our
/d
A a /•
A O ,C
KB,
d_
A/A-
C-
X?
(L.
t/*
PC,
(L,
c-
(L
ft*
(L.
(L-
A/A
CL.
(L.
(L-
C.
C
(L.
(L.
/>d.
(L.
01+4
A'/f
-------
Section No. 6
Revision No. 0
Date: March 14, 1990
Page 23 of 23
6.3.6 Data Analysis and Reporting
The objectives of data analysis and reporting are to summarize well construction characteristics,
well water treatment, farming practices, pesticide usage, and soil textures in the vicinity of the well,
and other information collected during the DWS Field Survey. This information will be merged with the
well water chemical analyses to determine how pesticide usage is related to groundwater vulnerability.
The procedures and quality assurance and quality control measures for data analysis and reporting
for domestic wells are similar to those for CWSs detailed in Section 5.2.7.
There are three additional quality control procedures available for the DWS that are not available
for the CWS. These include:
Debriefing of interviewers will provide valuable information on the ability of respondents
to understand and respond to particular questions; and
Comparing Second-stage County Agent Questionnaire responses with the DWS Local
Area questions to measure consistency.
• Post-stratifying NFS results to the number of rural wells on farmed and non-farmed
property by Census Region, to the value calculated in the 1987 American Housing
Survey.
6.3.7 Systems Security and Data Backup and Archival
The objective of systems security and data backup and archival is to establish and maintain a
system for the ongoing protection of data. The procedures and quality assurance and quality control
measures for DWS data are identical to those described for CWS data in Section 5.2.8.
-------
Appendix A
Revision No. 0
Date: March 14, 1990
Page 1 of 1
APPENDIX A
CWS Screening Survey Training Manual
-------
Appendix B
Revision No. 0
Date: March 14, 1990
Page 1 of 1
APPENDIX B
For blank copies of the
CWS Questionnaire,
CWS Team Leader Introduction and Well Observation Record, and
Local Area CWS Questionnaire
see Appendix D of the Phase I Report
-------
Appendix C
Revision No. 0
Date: March 14, 1990
Page 1 of 1
APPENDIX C
Calculation of Cluster Specifications
-------
Appendix D
Revision No. 0
Date: March 14, 1990
Page 1 of 3
APPENDIX D
STANDARD OPERATING PROCEDURES
1. TITLE: Development of CATI Questions
2. AREA OF RESPONSIBILITY: NFS DWS Well Screening Survey
3. GENERAL REQUIREMENTS:
a. Methodology
Westat TRC personnel will identify where urban areas of each county exist in relation to the target
areas by using Second-Stage County Cropped and Vulnerable Maps and a Rand McNally Zip Code
Atlas. TRC personnel will prepare county-specific questions to the identify the location of each
household in relation to the target and urban areas for the purpose of determining the eligibility of
each household for inclusion in the DWS Field Survey.
b. Equipment and Materials Required
i. Second-Stage County Cropped and Vulnerable Map
ii. Rand McNally Zip Code Atlas
4. PROCEDURE:
The Second-Stage County Cropped and Vulnerable Maps are received from ICF and logged in
the Second-Stage Map Receipt Check Sheet in the Mapping Workbook. These maps are then sent to
the Telephone Research Center (TRC) office in Frederick, MD. The remaining procedures are
executed by the TRC personnel using these Second-Stage County Cropped and Vulnerable Maps.
1. Obtain copies of detailed county maps from the county seat, city or town halls, and State
Department of Transportation. These maps should include highways and other landmarks.
2. Draw urban areas and major landmarks onto the second-stage maps using local maps of
the county. Complete the zip code checklist and develop the urban area and target area questions
for the CATI County-Specific Questions. The Zip Code Checklist includes the following steps:
(a) Draw zip code boundaries on the Second-Stage Map using the Rand McNally
Zip Code Atlas. Compare the computerized zip code list to the zip codes
found on the Rand McNally map.
(b) Resolve any discrepancies by calling the post office in the county or through
other appropriate action. Note any discrepancies found on the Zip Code
Check List.
(c) Indicate if the target area is included in the zip code area for each zip code in
the county.
-------
Appendix D
Revision No. 0
Date: March 14, 1990
Page 2 o< 3
(d) Determine if the county is urban or rural.
(e) Draw urbanized area(s) on the target map using census information.
(f) Estimate the percent of land covered by the target area and enter the
percentage on the bottom of the check list. Urban areas are not included in
the estimation.
(g) Forward the completed checklist to the DWS Survey Manager.
3. Write "city limit" questions to determine whether or not a household lies within the city limits
of any cities to be excluded from the Survey.
4. Write questions describing the boundaries of the urbanized areas to determine whether or
not a household lies within any urbanized areas in the county to be excluded from the Survey.
(Urbanized areas are defined by the Census Bureau as urban in character but not part of an
incorporated city or town.)
5. Overestimate rather than underestimate the urbanized area where the boundaries of the
urbanized area are unclear.
6. Identify zip codes within the county that are located within the target area.
7. Identify the physical boundaries of the target area within each zip code which contains a
target area using key landmarks such as rivers, mountains, highways, etc. Describe areas "between"
two or more landmarks rather than using compass directions.
8. Consult local county personnel, such as the County Planning Department, the County Clerk,
or other local county officials where landmark recognition may be questionable to determine whether
or not residents are familiar with the landmarks.
9. Travel to Westat's Twelve Oaks office on the day that the Zip Code Checklists are due for
specific counties. Attend the meeting with the other TRC personnel, the CATI Preparation Supervisor
and the DWS Survey Manager.
10. Review the Zip Code Checklists with the Map Boundaries Supervisor and the Mapping and
Cluster Specifications Manager. The final decisions regarding the smoothing of boundaries will be
made by the DWS Mapping and Cluster Specifications Manager.
11. Use the Zip Code Checklists to calculate the Cluster Specifications. The procedures for
calculating cluster specifications are described in Appendix C.
12. Review the urban area and target area questions for the CATI County-Specific Questions
with the other TRC personnel, the CATI Preparation Supervisor, the DWS Survey Manager and the
CATI Programming Supervisor. The CATI Preparation Supervisor will confirm that the TRC personnel
have verified the landmarks and roads that are to be used for the county-specific questions. Bring the
telephone log sheets to the meeting and give them to the CATI Preparation Supervisor for filing.
-------
Appendix D
Revision No. 0
Date: March 14, 1990
Page 3 of 3
13. Following approval, send the county-specific questions to CATI programming by the start
date for CATI Programming and Testing listed in the Master Schedule. Place a copy of the questions
in the log and give another copy of the questions to the DWS Survey Manager.
Upon completion of the CATI programming, the CATI Preparation Supervisor will log onto the VAX
and test the program by completing a dummy interview. This testing will be completed by the end
date listed on the CATI Programming and Testing Master Schedule. The testing will include every
path for the urban area and target area questions to ensure that the programming is correct. A check
mark will be placed beside each path after that path has been tested and its correctness verified
using a copy of the questions as a guide. The testing can be restricted to the urban area and target
area questions as the remainder of the CATI questions do not change from county to county.
-------
Appendix E
Revision No. 0
Date: March 14, 1990
Page 1 of 1
APPENDIX E
DOMESTIC WELL SCREENING
SURVEY TRAINING MANUAL
-------
Appendix F
Revision No. 0
Date: March 14, 1990
Page 1 of 1
APPENDIX F
DOMESTIC WELL SURVEY
INTERVIEWER TRAINING MANUAL
------- |